The potential of deploying large-scale antenna arrays in future wireless systems has stimulated extensive research on hybrid transceiver designs aiming to approximate the optimal fully-digital schemes with much reduced hardware cost and signal processing complexity. Generally, this hybrid transceiver structure requires a joint design of analog and digital processing to enable both beamsteering and spatial multiplexing gains. In this paper, we develop various weighted mean-square-error minimization (WMMSE) based hybrid transceiver designs over multiple-input multiple-output (MIMO) interference channels at both millimeter wave (mmWave) and microwave frequencies. Firstly, a heuristic joint design of hybrid precoder and combiner using alternating optimization is proposed, in which the majorization-minimization (MM) method is utilized to design the analog precoder and combiner with unit-modulus constraints. It is validated that this scheme achieves the comparable performance to the WMMSE fully-digital solution. To further reduce the complexity, a phase projection-based two-stage scheme is proposed to decouple the designs of analog and digital precoder combiner. Secondly, inspired by the fully-digital solutions based on the block-diagonalization zero-forcing (BD-ZF) and signal-to-leakage-plus-noise ratio (SLNR) criteria, low-complexity MMbased BD-ZF and SLNR hybrid designs are proposed to well approximate the corresponding fully-digital solutions. Thirdly, the partially-connected hybrid structure for reducing system hardware cost and power consumption is considered, for which the MM-based alternating optimization still works. Numerical results demonstrate the similar or superior performance of all the above proposed schemes over the existing benchmarks.

13 pages, 8 figures