The paper addresses the issue of the influence that the law of friction has on the dynamic behavior of the mechanical system that interacts with a tribosystem. The emergence of certain nonlinearities of higher order into the law of friction leads to an intensification of the dissipative process and to tribosystem destabilization. Consequently, friction excited self-oscillations are generated into the elements of the mechanical system with a wide spectrum of frequencies, sustained from the external source of energy. The theoretical and experimental modelling of the dissipative process and of the generation of is based on the frictional harmonic oscillator that interacts with the tribosystem. The oscillator is used as a sensitive element to the fluctuations of the frictional force and as a measure of the dissipated energy. Starting from the model, the elaboration of a method and of devices for experimental research provided the opportunity to study the behavior of the tribosystem in unstable operating. © Published under licence by IOP Publishing Ltd.