Quantitative susceptibility mapping (QSM) is a useful magnetic resonance imaging technique. We aim to propose a deep learning (DL)-based method for QSM reconstruction that is robust to data perturbations.We developed Diffusion-QSM, a diffusion model-based method with a time-travel and resampling refinement module for high-quality QSM reconstruction. First, the diffusion prior is trained unconditionally on high-quality QSM images, without requiring explicit information about the measured tissue phase, thereby enhancing generalization performance. Subsequently, during inference, the physical constraints from the QSM forward model and measurement are integrated into the output of the diffusion model to guide the sampling process toward realistic image representations. In addition, a time-travel and resampling module is employed during the later sampling stage to refine the image quality, resulting in an improved reconstruction without significantly prolonging the time.Experimental results show that Diffusion-QSM outperforms traditional and unsupervised DL methods for QSM reconstruction using simulation, in vivo and ex vivo data and shows better generalization capability than supervised DL methods when processing out-of-distribution data.Diffusion-QSM successfully unifies data-driven diffusion priors and subjectspecific physics constraints, enabling generalizable, high-quality QSM reconstruction under diverse perturbations, including image contrast, resolution and scan direction.This work advances QSM reconstruction by bridging the generalization gap in deep learning. The excellent quality and generalization capability underscore its potential for various realistic applications.