The aim of our work is to obtain the analytic solutions for a new nonlinear anharmonic oscillator by means of the Optimal Homotopy Asymptotic Method (OHAM), using only one iteration. The accuracy of the obtained results comes from the comparison with the corresponding numerical ones for specified physical parameters. Moreover, the OHAM method has a greater degree of flexibility than an iterative method as is presented in this paper. Based on these results, the analytically solutions of the Chen system were obtained for a special case (just one analytic first integral). The chaotic behaviors were excluded here. The provided solutions are usefully for many engineering applications.