Time‐resolved four‐wave mixing technique (FWM) is used to study the phonon induced dephasing of the excitons in CdTe self‐assembled quantum dots. The FWM signals show two different decay structures with fast and slow decay components, indicating a non‐Lorentzian lineshape with a zero‐phonon line and a broadband from exciton‐acoustic phonon interaction in the spectral region. The dephasing of the zero‐phonon line and the sideband components both show non‐linear characteristics with respect to temperature. The experimental data are reproduced well by a theoretical model taking into account the carrier‐acoustic phonon coupling.