The problem of signal detection for cognitive radar (CR) is considered, and a closed‐loop detection system is proposed to detect the extended target in the presence of signal‐dependent interference. In the system, the estimator of target impulse response (TIR) and waveform design are used to improve the detection performance of CR, and the estimate of TIR is also used for designing the matched waveform. In addition, in order to improve the reliability of detection, the two‐step process which is called estimation before detection is proposed. Simulation results indicate that the proposed system performs better than the traditional radar system which used the fixed waveform.