HSNNM: A Hebbian Sparse Neural Network Model for Efficient Transformer Learning
HSNNM: A Hebbian Sparse Neural Network Model for Efficient Transformer Learning
We introduce the Hebbian Sparse Neural Network Model (HSNNM), a biologically-inspired trans-former architecture that replaces the standard dense Feed-Forward Network (FFN) with a dynamicallyrouted system of sparse, competing expert sub-networks termed “lobes”. HSNNM integrates three coremechanisms: k-Winners-Take-All (k-WTA) lateral inhibition [4], non-gradient Hebbian plasticity, andnet2net structural neurogenesis. We demonstrate that HSNNM achieves a 49.6% reduction in FLOPsper token (7.18M vs 14.25M FLOPs/token) compared to a matched dense baseline, while incurring a 3.4percentage point accuracy trade-off (76.06% vs 79.45%), a favorable efficiency-accuracy balance giventhe significant compute savings. Through ablation studies, we show that both Hebbian plasticity andk-WTA sparsity contribute to stability over medium-term training (5,000 steps). Our results establishHSNNM as an efficient and robust alternative to dense transformers.
Transformer, Hebbian Learning, Net2Net, Efficient Deep Learning, Catastrophic Forgetting, Sparse Neural Networks, k-WTA
Transformer, Hebbian Learning, Net2Net, Efficient Deep Learning, Catastrophic Forgetting, Sparse Neural Networks, k-WTA
selected citations These citations are derived from selected sources.This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).0 popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.Average influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).Average impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.Average
Citations provided by BIP!Popularity provided by BIP!