Optimization✓ Mathematical
◆ The PatternAdjusting the learning rate over training for better convergence
A fixed learning rate is rarely optimal. Starting too high causes instability; finishing too high prevents convergence. Schedulers adjust α during training.
Cosine Decay: αₜ = αₘᵢₙ + ½(αₘₐₓ−αₘᵢₙ)(1 + cos(πt/T))
T = total steps | smoothly decays from αₘₐₓ to αₘᵢₙ
Warmup: α = αₘₐₓ · (t/t_warmup) for t < t_warmup
Linear ramp-up prevents large gradient updates from poorly-initialised weights
// Learning rate schedules — click to compare
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, T_max=100) scheduler = torch.optim.lr_scheduler.OneCycleLR(optim, max_lr=0.01, total_steps=1000) # call after each epoch/step: scheduler.step()
Pattern bridge: Cosine decay mimics natural cooling — fast changes early, fine adjustments later. Markets show it in sentiment cycles that heat up and cool down.