Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Preprint
Data sources: ZENODO
addClaim

PALA: Plateau-Aware Learning with Adaptive Memory Faster Convergence via SNR-Guided Plateau Classification and Cosine Bounce Restarts for Adam-Family Optimizers

Authors: Bayram, Cengizhan;

PALA: Plateau-Aware Learning with Adaptive Memory Faster Convergence via SNR-Guided Plateau Classification and Cosine Bounce Restarts for Adam-Family Optimizers

Abstract

PALA (Plateau-Aware Learning with Adaptive Memory) matches ReduceLROnPlateau in final accuracywhile using 25% fewer training epochs, reaching competitive accuracy 13 epochs sooner and early-stopping ata mean of 75 epochs versus 100 for all fixed-schedule baselines. This convergence efficiency is PALA’s primarypractical contribution: the same validation accuracy for a quarter less GPU compute, with no schedule-specifichyperparameter tuning.PALA achieves this by diagnosing why a model has stalled, not merely that it has stalled, and respondingwith a geometry-adapted escape strategy. It reads the first and second Adam moments already maintained bythe optimizer at zero additional cost: the gradient signal-to-noise ratio SNR = ∥E[g]∥/∥E[g2]1/2∥ and thegradient variance proxy GVar = E[∥v1/2t ∥] are jointly mapped through a threshold classifier to one of threelandscape categories (SADDLE, FLAT, or SHARP), each triggering a type-specific cosine learning-rate bounce. Aregion-of-interest (ROI) memory accumulates (floor_lr, loss_gain, plateau_type) triples acrossbounces; the next floor is selected by a recency–ROI weighted mean over type-matched entries. Adaptive β2scaling accelerates second-moment adaptation during bounces, and a β1 reduction on saddle exits increasesgradient direction reactivity. A gradient-agreement detector escalates bounce amplitude when the optimizerre-enters the same trap. On CIFAR-100 (PalaCNN) and STL-10, PALA outperforms plain Adam by +3.8 pp onaverage and matches ReduceLROnPlateau to within 0.2 pp in final accuracy. Our ablation further reveals thata more aggressive floor decay (γ=0.1) yields 63.3%, surpassing ReduceLROnPlateau (62.0%) by 1.3 pp.On ResNet-18, PALA underperforms in the 100-epoch regime due to BatchNorm’s suppression of gradientvariance signals.

Powered by OpenAIRE graph
Found an issue? Give us feedback