Abstract Motivated by the observation of in-situ nanocrystallization in the surface layer of plasma nitrided steels, this work is carried out to establish a thermodynamic model that describes the Gibbs free energy of BCC-Fe solution using the quasi-binary solution model. The model is specifically developed for the nitriding process of the multi-component steels, whose composition is first converted to an Fe-Creq-N ternary system using the concept of Cr equivalent through the relative elemental electronegativities with respect to nitrogen. With the thermodynamic model, the limit of stability of the BCC pseudo binary solution can be evaluated with temperatures and used to guide the selection of nitriding conditions. It has been recognized that the nitrogen-containing martensite is unstable and likely to decompose in the form of spinodal decomposition over a specific compositional/temperature range. 40CrNi steel is adopted in this work and plasma nitrided at 800 K for 8 h. Nano-scale FeN1/12 phase and high‑nitrogen martensite phase have been experimentally observed in the nitrided surface layer, which embraces the thermodynamic predictions in terms of nanocrystallization via spinodal decomposition.