The quadratic time and memory complexity of standard self-attention mechanisms (O(N2)) imposes fundamental computationallimits on scaling Transformer context windows. In this work, we introduce SpectralBERT, an efficient bidirectionalarchitecture that replaces standard pairwise token attention with a fixed-size spectral memory mechanism. By projectingsequence embeddings into the frequency domain via Real Fast Fourier Transform (RFFT), adaptively pooling frequencycoefficients into compact slot representations, and retrieving context via memory-guided attention, SpectralBERT achieves O(Nlog N) time complexity with constant sequence memory overhead (O(1)) during feature mixing. Empirical evaluations onWikiText-2 demonstrate a final Masked Language Modeling (MLM) loss of 5.6420, outperforming a comparably sized BERT-Mini baseline (loss 6.5773) by 14.3% while training 1.13x faster. Pre-training on WikiText-103 at an 8,192-token contextlength yields a loss of 5.2525 (Perplexity = 191.05). Forward execution scaling benchmarks confirm a 14.51x throughputspeedup at a 65,536-token sequence length on a single NVIDIA Tesla T4 GPU. These findings demonstrate that frequency-domain memory structures offer a scalable and mathematically sound alternative to quadratic self-attention for long-contextlanguage modeling.