Abstract The prior austenite grain (PAG) size of a low alloy high strength martensitic steel, AF9628, was refined by rapid austenitizing heat treatments at 825 and 850 °C. The reduction in PAG size of as high as 97% was obtained. After one rapid austenitizing cycle, the newly nucleated (small) PAGs were observed to be preferentially located at the prior austenite grain boundaries of starting microstructure, although a small number of fine PAGs were also observed in the interior of PAGs of starting microstructure. A more uniform distribution of fine PAGs was observed with an increase in the number of cycles. The average number of austenite nucleation sites in a given area was 43.4% higher at 850 °C than 825 °C after one cycle. A higher nucleation rate at 850 °C resulted in a smaller PAG size at this temperature than 825 °C after one and four cycles. A higher growth rate at 850 °C resulted in a larger PAG size at this temperature than 825 °C after sixteen cycles. Prior studies have discussed the role of growth rate in microstructure evolution during rapid austenitizing heat treatments at different temperatures and not the role of nucleation rate, which is discussed for the first time in current study. Furthermore, the X-ray diffraction results indicated a higher amount of retained austenite in the finest microstructure and it can affect the strength in these microstructures. This aspect of microstructure evolution is also not reported in prior studies on microstructural refinement with rapid austenitizing method.