An unscented Kalman filter (UKF), which is a kind of new style filter based on unscented transform, has been used to estimate spacecraft orbit with real magnetometer. In order to simulate the measurements, this paper has established the high-accuracy dynamic model of the orbit and the 10th-order model of the geomagnetic field. Two navigation algorithms are presented respectively. Firstly, the orientation and the magnitude of the geomagnetic field are chosen as the filer's observation. Given a magnetometer with a 200nT 3-/spl sigma/ accuracy, the system can achieve 3-/spl sigma/ position accuracy on the order of 2 km, secondly, when only the magnitude is chosen, system can achieve position accuracy on the order of 2.5 km. The results also demonstrate that the UKF method has better convergence ability and higher accuracy than the extended Kalman filter (EKF) method.