Reproducing the electronic structure of AM4X8 lacunar spinels with a breathing pyrochlore lattice is a great theoretical challenge due to the interplay of various factors. The character of the M4X4 cluster orbitals is critically influenced by the Jahn-Teller instability, the spin-orbit interaction, and also by the magnetic state of the clusters. Consequently, to reproduce the narrow-gap semiconducting nature of these moderately correlated materials requires advanced approaches, since the strength of the intercluster hopping is strongly affected by the character of the cluster orbitals. In order to provide a solid experimental basis for theoretical studies, we performed broadband optical spectroscopy on a large set of lacunar spinels, with systematically changing ions at the A and M sites as well as the ligand (A = Ga, Ge, Al; M = V, Mo, Nb, Ta; X = S, Se). Our study covers the range of phonon excitations and also electronic transitions near the gap edge. In the phonon excitation spectrum a limited subset of the symmetry-allowed modes is observed in the cubic state, with a few additional modes emerging upon the symmetry-lowering structural transition. All the infrared-active modes are assigned to vibrations of the ligands and ions at the A sites, with no obvious contribution from the M-site ions. Concerning the electronic states, we found that all compounds are narrow-gap semiconductors (Eg=130-350 meV) already in their room-temperature cubic state and their structural transitions induce weak, if any, changes in the band gap. The gap value is decreased when substituting S with Se and also when replacing 3d ions by 4d or 5d ions at the M sites.