Since the optimal codebook is channel-dependent, it is crucial to design a codebook which is suitable for a certain channel condition, such as Gaussian, Rayleigh and Rician fading channels. In Chapter 2, the key performance indicators (KPIs) of the SCMA codebook are highlighted and the state-of-the-art codebook design schemes are also presented. In Chapter 3, a novel multi-task learning empowered end-to-end SCMA (E2E-SCMA) framework is introduced for joint SCMA encoding and detection optimization in Gaussian channels. Building upon a new SCMA mapping design with linear encoding, an efficient SCMA encoder is proposed, which can reduce the depth of the network and thereby help prevent the gradient from vanishing. At the receiver side, each user is viewed as a single learning task and then the decoder is designed in a task-specific fashion. Chapter 4 investigates the low-projection codebook (LPCB) design in downlink Rician fading channels, which is motivated by the strong need to provide ultra-low decoding complexity and good error performance for a massive number of low-end and low-cost Internet-of-things (IoT) communication devices. Novel codebook design criteria are derived for downlink Rician fading channels using pair-wise error probability (PEP). To reduce the decoding complexity of message passing in SCMA, one possible approach is to reduce the effective number of constellation points, which is the key principle of the proposed LPCBs. Chapter 5 aims at enhancing the signal space diversity of SCMA by introducing quadrature component delay to the transmitted codeword of in downlink Rayleigh fading channels. Such a system is called SSD-SCMA. By studying the average mutual information (AMI) and the PEP of the proposed SSD-SCMA, we develop novel codebooks by maximizing the derived AMI lower bound and a modified minimum product distance (MMPD), respectively. The intrinsic asymptotic relationship between the AMI lower bound and proposed MMPD based codebook designs is also revealed.

Existing SCMA codebooks mainly design SCMA codebooks based on a regular structure with same codebook size. Nevertheless, this implicitly assumes that the SCMA users sharing the same radio resources have a similar quality of service (QoS) requirement and channel condition, which may not be realistic in practical scenarios. To address this issue, Chapter 6 proposes a variable modulation SCMA (VM-SCMA) system and an adaptive VM-SCMA (AVM-SCMA) system for uplink Rayleigh fading channels. The proposed VM-SCMA allows users to employ codebooks with different modulation orders by minimizing the error rate performance of the worst user. In addition, an adaptive VM-SCMA (AVM-SCMA) scheme is designed by maximizing the effective throughput of VM-SCMA subject to a reliable error rate constraint.

The ever-increasing demand for higher data rates, improved spectral efficiency, and massive connectivity has driven the rapid evolution of wireless communication systems. To meet these requirements, sparse code multiple access (SCMA) technique has been envisioned as a promising technique for future wireless communication networks. A fundamental research problem in SCMA is how to design efficient sparse codebook for low error rate transmissions, which remains an open research problem.