The recent evolution in Neural Networks has led to the introduction of a new promising Network known as the Spiking Neural Network (SNNs) which is supposed to be energy-efficient due to the fact that it only spikes fire over time. Practically when these networks are trained with surrogate gradients, their activities often become unstable, either almost all neurons fire too much (“epileptic saturation”) or almost none fire at all (“comatose silence”) due to non-differentiable nature of spike generation. In this study we investigate the efficacy of Homeostatic L2 Regularization in enforcing sparsity constrains on a rate-coded RSNN for temporal anomaly detection. Homeostatic L2 regularization technique pushes the average firing rate towards desired target of 5% and theoretical this should keep the RSNN sparse. However, the experiment shows otherwise and the network settles into a high-actively regime with about 49.2% of neurons firing while it achieves a 100% classification accuracy on the task, which implies that the network solves the problem but in a very dense, power-hungry way, more like the conventional digital circuit than a sparse brain-like SNN. However, it is suggested that the gradient descent landscape for rate-coded RSNNs prioritizes noise robustness (maximum signal-to-noise ratio via saturation) over metabolic efficiency, rendering soft regularization insufficient. Such that the “Synaptic Efficiency” metric was adopted to quantify how much useful computation or information transfer is achieved per unit of the synaptic activity. With this metric it fell out that networks can be accurate yet synaptically inefficient when they operate in saturated regimes. Based on this observation it was concluded that achieving truly neuromorphic sparity cannot relay less-functional penalties alone. Instead, the architecture must include hard constraints like enforced refractory periods, strict limits on firing, or other built-in mechanisms so as to structurally prevent the network from drifting into high-firing, energy-expensive modes while training.