A magnetically-quiet environment is important for detecting faint magnetic-field signals or nonmagnetic spin-dependent interactions. Passive magnetic shielding using layers of large magnetic-permeability materials is widely used to reduce the magnetic-field noise. The magnetic-field noise can also be actively monitored with magnetometers and then compensated, acting as a complementary method to the passive shielding. We present here a general model to quantitatively depict and optimize the performance of active magnetic-field stabilization and experimentally verify our model using optically-pumped atomic magnetometers. We experimentally demonstrate a magnetic-field noise rejection ratio of larger than ∼800 at low frequencies and an environment with a magnetic-field noise floor of ∼40 fT/Hz1/2 in unshielded Earth’s field. The proposed model provides a general guidance on analyzing and improving the performance of active magnetic-field stabilization with magnetometers. This work offers the possibility of sensitive detections of magnetic-field signals in a variety of unshielded natural environments.