Hyperspectral imaging is a valuable technique for accurately classifying materials because of the abundance of spectral information and high resolution it provides. However, the characteristics of Hyperspectral Imaging, such as high-dimensional features and information redundancy, pose significant challenges to data processing. Traditional dimensionality reduction methods often have information loss, high computational complexity, and easy to ignore the strong correlation between HSI bands when dealing with the HSI data. Although other methods can achieve satisfactory classification performance, they do not consider the dimensionality reduction of HSI, and they focus on the model performance, which limits further improvement in classification performance. This article proposes a transformer-based framework called “SpectrumRecombineFormer” (SRF), which is composed of two key modules, namely “Spatial–Spectral Recombination” (SSRC) and “Cross-Layer Fusion” (CF). The SSRC is capable of utilizing both adjacent and non-adjacent spectrums to generate the spatial-sequential perceptive representations, which alleviate the effect of the strong correlation between HSI bands. The CF can avoid the loss of information during the feed-forward procedure among layers. Extensive experiments on five existing datasets (widely adopted Indian Pines, Houston2013, Pavia University, Salinas, and KSC) demonstrate the capability of our proposed method to address the above-mentioned challenges. Both quantitative and qualitative experimental ablation studies, including visualization results, reveal that the proposed SRF method can successfully and efficiently classify HSIs and surpass the other state-of-the-art methods. For access to the source code, please visit https://github.com/kangpeilun/SRF-HSI-Classification-master .