The objective of this work is to highlight and discuss some critical conditions in which the imprecision of atomic force microscopy (AFM) system will be intensified. To precisely address these issues, we developed a complete close form solution for a non-linear motion of AFM system subjected to non-linear contact and van der Waals forces. Galerkin method and multiple time scale approach are used to solve the governing non-linear differential equation of the AFM system. A nonlinear frequency-response equation is then obtained as a function of displacement excitation, resonance frequency and associated response amplitude. By simulation of the AFM response under different tip-sample interaction force and distance, it is shown that jump phenomenon takes place due to the non-linear motion of microcantilever at certain operating frequencies. This case can be avoided via operating the AFM in a right frequency region; however the error associated with it must be compensated by a post-processing of collected data.