We report on the structural and spectroscopic characterization of the multiferroic Fe2Mo3O8. Synchrotron X-ray and neutron diffraction as well as thermal expansion measurements reveal a lattice anomaly at TN≃60K but do not show any symmetry lowering in the magnetically ordered state. The lattice parameter c exhibits a nonmonotonic behavior with a pronounced minimum around 200 K, which is also reflected in an anomalous behavior of some of the observed infrared-active optical excitations and parallels the onset of short-range magnetic order. The infrared reflectivity spectra measured between 5 and 300 K in the frequency range of 100-8000cm-1 reveal most of the expected phonon modes in comparison with the eigenfrequencies obtained by density-functional calculations. The A1 phonons show an overall hardening upon cooling, whereas a nonmonotonic behavior is observed for some of the E1 modes. These modes also show a strongly increased phonon lifetime below TN, which we associate with the quenched direction of the orbital moment in the magnetically ordered state. A similar increase is observed in the lifetime of the higher-lying d-d excitations of the tetrahedral Fe2+ site that become clearly visible below TN only.