Abstract The stress–strain curves and yield loci of metallic materials under biaxial tensile tests can be precisely predicted using a yield criterion. In this study, biaxial tensile testing equipment with high-accuracy displacement, temperature control, and digital speckle correlation method of non-contact optical strain measurement is developed to study the thermo-mechanical deformation behavior of a GH738 nickel-based superalloy, which is widely used for important components of aeronautic engines, to exploit its excellent mechanical properties at high temperatures. Additionally, the used specimen shape is discussed. The results of this study show that the special sample shape size solves the problem in which an extremely thin sheet metal bulges when subjected to tension. The equivalent stress–strain curves and yield surfaces, which were fitted according to the Hill'48 yield criterion, exhibit deformation characteristics that are more precise than those according to the von Mises criterion. Eventually, the efficacy of the related test methods is corroborated.