In this paper, the potential benefits of applying non-orthogonal multiple access (NOMA) technique in $K$-tier hybrid heterogeneous networks (HetNets) is explored. A promising new transmission framework is proposed, in which NOMA is adopted in small cells and massive multiple-input multiple-output (MIMO) is employed in macro cells. For maximizing the biased average received power for mobile users, a NOMA and massive MIMO based user association scheme is developed. To evaluate the performance of the proposed framework, we first derive the analytical expressions for the coverage probability of NOMA enhanced small cells. We then examine the spectrum efficiency of the whole network, by deriving exact analytical expressions for NOMA enhanced small cells and a tractable lower bound for massive MIMO enabled macro cells. Lastly, we investigate the energy efficiency of the hybrid HetNets. Our results demonstrate that: 1) The coverage probability of NOMA enhanced small cells is affected to a large extent by the targeted transmit rates and power sharing coefficients of two NOMA users; 2) Massive MIMO enabled macro cells are capable of significantly enhancing the spectrum efficiency by increasing the number of antennas; 3) The energy efficiency of the whole network can be greatly improved by densely deploying NOMA enhanced small cell base stations (BSs); and 4) The proposed NOMA enhanced HetNets transmission scheme has superior performance compared to the orthogonal multiple access~(OMA) based HetNets.

IEEE Journal on Selected Areas in Communications (JSAC), 2017