Abstract A new criterion is presented for prediction of intact rock and rock mass failure under polyaxial state of stresses. A comprehensive database of the results of uniaxial, triaxial, and polyaxial tests on intact rock was utilized for evaluation of the new criterion and comparison of its accuracy with the most accurate and frequently used criteria. Analysis of 195 individual data groups of brittle failure in the form of (σ3,σ1) for twelve different rock types showed that the proposed criterion lead to determination coefficients higher than 0.99 in most cases. It also gave the lower values of root mean squared errors relative to Mohr–Coulomb and Hoek–Brown criteria in fitting the normalized strength data for each rock type. The criteria were used to fit a typical series of triaxial strength data including brittle and ductile behavior. It showed that the new criterion can maintain its accuracy over a wider range of stresses. In the absence of rock mass strength data, applicability of the new criterion for rock mass was verified by fitting it to typical Hoek–Brown failure envelopes. Regression analysis of the polyaxial strength data in the form of (σ3,σ2,σ1) for six rock types showed that the new criterion predict the strength more accurately than the Modified Wiebols–Cook and You criteria in all cases.