Purpose: To investigate the feasibility of using an image‐based method to correct for spectral distortions from photon‐counting detectors for their application in breast computed tomography (CT). Methods: The polyenergetic incident spectrum was simulated with the tungsten anode spectral model. Experiments were performed on a Cadmium‐Zinc‐Telluride (CZT) photon‐counting detector with five energy thresholds. BR12 phantoms of various thicknesses were used for calibration. A non‐linear function was selected to fit the count correlation between the simulated and the measured spectra in the calibration process. To evaluate the proposed spectral distortion correction method, both the corrected counts and the effective attenuation coefficients were compared to the simulated values for polymethyl methacrylate (PMMA) phantoms of 8.7 mm, 48.8 mm and 100.0 mm. The feasibility of applying the proposed method to quantitative material decomposition was tested using a dual‐energy imaging technique with a three‐material phantom that consisted of water, lipid and protein. Results: The implementation of the proposed method reduced the relative RMS error of the output counts in the five energy bins with respect to the simulated incident counts from 23.0%, 33.0% and 54.0% to 1.2%, 1.8% and 7.7% for 8.7 mm, 48.8 mm and 100.0 mm PMMA phantoms, respectively. The accuracy of the effective attenuation coefficient of PMMA estimate was also improved with the proposed spectral distortion correction. Finally, the relative RMS error of water, lipid and protein decompositions in dual‐ energy imaging was significantly reduced from 53.4% to 6.8% after correction was applied. Conclusions: The study demonstrated that the proposed method can effectively reduce the spectral distortions caused by various artifacts, including pulse pileup and charge sharing effects. It may be used as a generalized procedure for the spectrum distortion correction of different photon‐counting detectors in clinical breast CT systems.