One presents how the quantum dynamics of the optical cooling of an opto-mechanical system are affected by a secular approximation. A laser driven two-level quantum dot (QD) embedded in a phononic nano-cavity is investigated for moderately strong QD-phonon couplings regimes. For these regimes, the use of a secular approximation within the QD-phonon interaction terms is no longer justified as the rapidly oscillating terms cannot be neglected from the system dynamics. Therefore, one shows that although being small, their contribution plays an important role when quantum cooling is achieved. The main contribution of the fast oscillating terms is analytically estimated and one compares how the quantum cooling dynamics change within or beyond the secular approximation. The behavior of the quantum cooling effect is investigated in the steady-state regime via the phonon field statistics.