In this study, average symbol error rate (ASER) expressions for rectangular quadrature amplitude modulation (RQAM) and cross quadrature amplitude modulation (XQAM) over kappa - mu shadowed fading are proposed by considering two different system models. In the first system model, a single-input-single-output (SISO) scheme is considered while a multiple-input-multiple-output (MIMO) with single transmit antenna selection and maximal ratio combining (TAS/MRC) is investigated as a second system model. First, the exact, approximate and asymptotic ASER expressions of RQAM and XQAM for a SISO system model are proposed based on probability density function (PDF) of instantaneous signal-to-noise ratio (SNR). Second, ASER expressions of RQAM and XQAM schemes for a MIMO system with TAS/MRC are obtained by using the PDF of instantaneous SNR at the receiver side. Then, ASER performances of RQAM and XQAM for the considered two system models are compared in the presence of different fading configurations. Finally, our numerical results illustrate that the analytical ASER expressions proposed in this study are well agreement with their simulations. (C) 2019 Elsevier B.V. All rights reserved.