Silicon drift detector with high sensitivity and energy resolution is an advanced detector which is suitable to be used in deep space detection. To study and reveal the radiation damage of the silicon drift detector (SDD) in a deep-space environment, which will degrade the detector performance, in this paper, the SDD radiation damage effects and mechanics, including displacement damage and ionization damage, for irradiations of different energy of neutrons and gammas are investigated using Geant4 simulation. The results indicate the recoil atoms distribution generated by neutrons in SDD is uniform, and recoil atoms’ energy is mainly in the low energy region. For secondary particles produced by neutron irradiation, a large energy loss in inelastic scattering and fission reactions occur, and neutron has a significant nuclear reaction. The energy deposition caused by gammas irradiation is linear with the thickness of SDD; the secondary electron energy distribution produced by gamma irradiation is from several eV to incident particle energy. As the scattering angle of secondary electron increases, the number of secondary electrons decreases. Therefore, a reasonable detector epitaxial thickness should be set in the anti-irradiation design of SDD.