The accumulated plastic strain energy density at a dangerous point is studied to estimate the low cycle fatigue life that is composed of fatigue initiation life and fatigue crack propagation life. The modified Ramberg–Osgood constitutive relation is applied to characterize the stress–strain relationship of the strain-hardening material. The plastic strain energy density under uni-axial tension and cyclic load are derived, which are used as threshold and reference values, respectively. Then, a framework to assess the lives of fatigue initiation and fatigue crack propagation by accumulated plastic strain energy density is proposed. Finally, this method is applied to two types of aluminum alloy, LC9 and LY12 for low-cycle fatigue, and agreed well with the experiments.