To achieve full-duplex wireless communication, the most essential issue is to eliminate in-band self-interference caused by simultaneous transmission and reception. In this paper, we investigate the impact of transmitter IQ imbalance on digital signal processing (DSP)-assisted analog self-interference cancellation, where two transmitters are used for transmission and cancellation, respectively. Our result implies that the transmitter IQ imbalance results in residual self-interference after conventional DSP-assisted analog cancellation, which still has a much higher power compared to the desired signal. Then, a baseband cancellation signal generated by widely linear (WL) filtering is proposed to effectively handle the transmitter IQ imbalance, and the optimal WL filter and the adaptive algorithm to obtain them can be derived. The impact of receiver IQ imbalance on the proposed adaptive algorithm is analyzed, and we show that the algorithm is capable of obtaining the same optimal WL filter despite receiver IQ imbalance. In addition, for practical implementation, we propose an adaptive algorithm based on the augmented complex least mean squares (ACLMS) method to obtain the WL filter, where no information about the transmitter and receiver IQ imbalances is needed. The steady state behaviors of the proposed adaptive algorithms are analyzed and the performance bounds are derived. Numerical simulation results confirm the validity and superior performance of the proposed ACLMS algorithm in various practical scenarios of transmitter and receiver IQ imbalances. Also, the tolerance of the proposed algorithm to other transceiver impairments was investigated using simulations.