A fracture criterion is newly proposed to evaluate fracture behavior and predict fracture initiation of metal materials in different complicated stress states for four different fracture mechanisms including quasicleavage fracture, normal fracture with void, shear fracture with void, and shear fracture without void. The dominant factors of these four different mechanisms are distinct, so it is impossible to capture all features of fracture initiation under different stress states with a single criterion, and different functions are necessary to predict fracture initiation of different mechanisms. In the new fracture criterion, different branches of the fracture criterion have been proposed corresponding to different fracture mechanisms. Quasicleavage fracture and normal fracture with void are described as a function of the principal stress, shear fracture with void is a function of the stress triaxiality and maximal shear stress, and shear fracture without void is only controlled by the maximal shear stress. The new fracture criterion is applied to predict the fracture initiation site and the fracture direction of nodular cast iron QT400-15 in combined tension-torsion tests. Predicted results are compared with experimental results to validate the performance of the new criterion in the intermediate stress triaxiality between 0 and 1/3. The new criterion is also applied to predict the crack initiation site and the direction of crack initiation of LY12 aluminium alloy and HY130 mild steel in mixed mode fracture tests to validate the performance of the new criterion in the high stress triaxiality. The new fracture criterion gives consistent results for these materials in a wide stress triaxiality range. It is shown that the new fracture criterion is a better supplement to the deficiency of fracture mechanics and also a better amendment to traditional strength theory in complicated stress states. Therefore, the new fracture criterion is recommended to be utilized to evaluate the fracture initiation of metal structures in nuclear waste storage and other engineering applications.