This paper deals with the issues on the optimization of transmitter and receiver FIR filters in the orthogonal frequency division multiplex (OFDM) system with offset QAM modulation. The filters have their responses overlapped in both the time and the frequency domains. The transmitter and receiver filters satisfy the constraints minimizing the interchannel and intersymbol interference and are optimized by using nonlinear programming procedures. Two different objective functions in the frequency domain are used: the least square error (LSE) in the stop-band and minimizing the maximum magnitude (mini-max) in the stop-band.