Abstract Due to broad usage of anisotropic composite beams in modern engineering structures, the main goal of the present work is to examine their geometrically nonlinear vibration. To this end, a third–order shear deformation theory with a nonlinear von-Karman strain field is used for anisotropic beams and combined with the advantages of the isogeometric framework. The layup properties are assumed to be anisotropic in the depth direction and a transient tip follower force is considered. The governing nonlinear equations of vibration are integrated by means of the Newmark approach and solved with the Newton-Raphson method. Flutter loads as well as natural frequencies are obtained by eigenvalue analysis. The effects of various important factors such as non-prismatic shape, orientation of the composite fibers, critical follower force and bifurcation point are studied, using both h- and p-refinements. The results show that the nonlinear vibration and flutter characteristics of the anisotropic composite beams are completely different from those for orthotropic and isotropic ones. Thick beams with anisotropic layups are more sensitive to the shear parameter than conventional ones and deform primarily in shear mode rather than in bending. Anisotropic beams reveal a higher flutter instability force than other cases for a given shear parameter value. Another important phenomenon is that the stress distribution in anisotropic layups shows irregular patterns both in depth and time. Anisotropic layups with ply angles between 15 ∘ and 45 ∘ seem to present enhanced nonlinear performance with respect to other layup choices. A coarse mesh of quartic C 3 B-splines is observed to provide high accuracy for nonlinear deflections in anisotropic cases even for rather low shear parameters.