This thesis proposes a new Anisotropic Matsuoka-Nakai (AMN) criterion to characterize the failure of transversely isotropic rocks under true triaxial stress states. One major obstacle in formulating an anisotropic criterion is that it usually involves six stress components, instead of three principal stresses. As such, anisotropic criteria usually lead to complicated mathematical expressions, and cannot be directly visualized in three-dimensional space. This problem is solved by introducing the Material Normal Stress System (MNSS), which is the space formed by the three normal stress components reflecting the material anisotropy. Within this system, the failure behavior of transversely isotropic rocks in conventional triaxial tests can be represented with geometrical features in the MNSS. These features are then incorporated into the failure surface of the original Matsuoka-Nakai criterion in the Material Normal Stress System, resulting in the Anisotropic Matsuoka-Nakai criterion. This criterion, combined with the Coulomb criterion, is validated against both conventional and true triaxial test data, that are collected from an extensive literature review. The combination of the AMN criterion and the Coulomb criterion satisfactorily characterizes the measured strength from an extensive program of true triaxial tests on a schist, which confirms the ability of the proposed criterion. Finally, this combination of criteria is applied to the borehole stability problem. The necessary mud pressure against borehole collapse and the onset of borehole failure are examined.