Abstract A nonlinear finite element method, based on the von Karman-High Order Shear Deformation Theory (HOST) and on the principle of minimal total potential energy, is used for the analysis of buckling and post buckling behavior and of the first-ply failure of an axially compressed laminated type composite structure. For this purpose and for the analysis of the instability responses an improved 4-node layered shell finite element is proposed. The finite element formulation is based on the third order shear deformation theory with four nodes shell elements having eight degrees of freedom per node. The first-ply failure of laminates and the delamination are some of the features incorporated in the geometric nonlinear formulation. The load-displacement relations for different types of graphite/ epoxy laminates are obtained. Stresses are computed in order to determine the first-ply failure of the mentioned axially compressed laminated composite structure. The buckling, post buckling and failure behavior of axially compressed curved composite panels are investigated numerically and compared with the experimentally obtained results. The buckling loads using linear and geometrically nonlinear analyzes are compared with experimental results and a good agreement between them is obtained. It is shown that the effect of stacking sequence on buckling load is evident.