Centrifugal pump is a key part of nuclear power plant systems, and its health status is critical to the safety and reliability of nuclear power plants. Therefore, fault diagnosis is required for centrifugal pump. Traditional fault diagnosis methods have difficulty extracting fault features from nonlinear and non-stationary signals, resulting in low diagnostic accuracy. In this paper, a new fault diagnosis method is proposed based on the improved particle swarm optimization (IPSO) algorithm-based variational modal decomposition (VMD) and relevance vector machine (RVM). Firstly, a simulation test bench for rotor faults is built, in which vibration displacement signals of the rotor are also collected by eddy current sensors. Then, the improved particle swarm algorithm is used to optimize the VMD to achieve adaptive decomposition of vibration displacement signals. Meanwhile, a screening criterion based on the minimum Kullback-Leibler (K-L) divergence value is established to extract the primary intrinsic modal function (IMF) component. Eventually, the factors are obtained from the primary IMF component to form a fault feature vector, and fault patterns are recognized using the RVM model. The results show that the extraction of the fault information and fault diagnosis classification have been improved, and the average accuracy could reach 97.87%.