A single-beam spin-exchange relaxation-free atomic magnetometer is ultra-sensitive in the zero field, which has great potential for the detection of a magnetoencephalogram. The addition of a modulated magnetic field is an important approach to achieve high sensitivity for devices of this kind. In this study, we discovered that the amplitude and frequency of the modulated magnetic field (modulation index 0–3) both influence the light absorption. We defined this effect into a function by combining theoretical analysis and the results of experiments. It is discovered that the transmission intensity decreases with an increase in the modulation index. This effect is weakened under the application of a high modulation index. In addition, the transmission intensity and bias magnetic field no longer follow a strict Lorentz curve, while a high degree of fit can be achieved by applying the numerical solution of the Bloch function. A compact magnetometer with a volume of 10 cm3 and a sensitivity of 20 fT/Hz is developed based on the single beam scheme for the proof of concept. Our study is crucial in two aspects: (1) Obtaining high sensitivity through a short measurement period and (2) alignment of the scale factor of the individual magnetometer in a detection array, which further pave the way for improvement in a magnetometer’s performance under a variety of optics platforms.