Crystal of GaV4S8, a multiferroic system hosting a Néel-type skyrmion lattice phase, has been investigated by polarized Raman and IR spectroscopy above and below the ferroelectric phase transition. Counts of the observed IR and Raman-active modes belonging to distinct irreducible representations agree quite well with group-theory predictions. Phonon spectra are assigned and interpreted with the aid of ab initio calculations of the phonon spectra in the ferroelectric phase. Results allow appreciation of phonon frequencies of the modes involved in Jahn-Teller distortion and their contribution to the spontaneous polarization.