Resonant Inelastic X-Ray Scattering (RIXS) is a well-established tool for probing excitations in a wide range of materials. The measured spectra strongly depend on the scattering geometry, via its influence on the polarization of the incoming and outgoing light. By employing a tensor representation of the 4-point response function that governs the RIXS intensity, we disentangle the experimental geometry from the intrinsic material properties. In dipole-dipole RIXS processes and low-symmetry crystals, up to 81 linearly independent fundamental spectra can be measured as a function of light polarization. However, for crystals or molecules with symmetry, the number of independent fundamental spectra that define the RIXS tensor is significantly reduced. This work presents a systematic framework for determining the number of fundamental spectra and expressing the RIXS tensor in terms of these fundamental components. Given a specific experimental geometry, the measured spectrum can be represented as a linear combination of these fundamental spectra. To validate our approach, we performed calculations for different point group symmetries, both with and without an applied magnetic field. Within the same framework, we derived expressions for powder spectra in momentum-independent processes and spectra obtained using Bragg spectrometers. This formalism provides a valuable toolkit for optimizing experiment planning, data interpretation, and RIXS simulation.