A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
Paper
• 2102.06356 • Published
Training
Note Optimizer-Google
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Paper • 1904.00962 • Published • 1Note Optimizer-lamb
Decoupled Weight Decay Regularization
Paper • 1711.05101 • Published • 4Note Optimizer-adamw https://arxiv.org/abs/2410.05192 Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective Stanford. WSD to LR