Cyclical Learning Rates
Cyclical learning rates repeatedly cycle the learning rate up and down between a lower and upper bound instead of only decaying it.
Overview
This counterintuitive bouncing can speed up convergence and helps the optimizer escape sharp local minima and saddle points.
Deep Dive
Proposed by Leslie Smith in 2015, cyclical learning rates (CLR) challenge the assumption that the rate should only ever decrease. Instead, it oscillates between a minimum and maximum bound over a fixed number of iterations (a 'cycle'), often with a triangular shape. The intuition: periodically raising the rate provides a burst of energy that lets the model jump out of poor, sharp minima and traverse saddle points, while the low phases let it settle. Smith also introduced the 'LR range test' — a short run that sweeps the rate upward while watching the loss — to find good bounds automatically. Triangular, triangular-with-decay, and the famous one-cycle policy all build on this idea.
Technical Insight
A triangular policy linearly increases the rate from a base to a max over half a cycle, then linearly decreases it back over the other half. The cycle length is usually set to a few epochs' worth of iterations. The one-cycle policy uses a single long cycle: rate rises then falls below the starting point, while momentum moves inversely — high when the rate is low and vice versa — which acts as a regularizer and enables 'super-convergence' on some tasks.
Strategic Impact
Cost and budget
Architecture decisions drive performance and operating cost for years.
Clearer decisions
Technical education helps teams choose the right stack, not just the newest one.
Quality control
Better engineering choices reduce reliability incidents in production.
The Future of Cyclical Learning Rates
Cyclical schedules and the one-cycle policy remain popular for fast training on vision and tabular tasks, and the LR range test is a standard tuning trick. For very large language models, smooth warmup-plus-cosine schedules tend to dominate, but the underlying insight — that strategic increases help escape bad regions of the loss landscape — informs warm restarts (SGDR) and ensemble methods that snapshot models at each cycle's low point. Expect continued cross-pollination between cyclical ideas and adaptive, self-tuning schedulers.
Real-World Implementation
fast.ai popularized the one-cycle policy as a default for quickly training image classifiers to high accuracy in few epochs.
The LR range test sweeps the rate upward over a few hundred batches to pick min and max bounds before a real run.
Snapshot ensembling saves a model checkpoint at the end of each cycle, producing a free ensemble from one training run.
Stochastic Gradient Descent with Warm Restarts (SGDR) periodically resets the rate to a high value to escape sharp minima.
Risks & Guardrails
Optimizing one benchmark can hide broader system weaknesses.
Infrastructure and maintenance costs are often underestimated.
Security and observability gaps can grow as systems become more complex.
Implementation Roadmap
Define latency, quality, and cost targets before implementation.
Benchmark under realistic load and data conditions.
Instrument monitoring for errors, drift, and user impact.
Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Cyclical Learning Rates quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
Learning Rate Scheduling
Frequently asked questions
What is Cyclical Learning Rates?
Cyclical learning rates repeatedly cycle the learning rate up and down between a lower and upper bound instead of only decaying it. This counterintuitive bouncing can speed up convergence and helps the optimizer escape sharp local minima and saddle points.
What core assumption do cyclical learning rates challenge?
CLR deliberately raises the rate again and again, rejecting the traditional view that it must only decay monotonically.
Why might periodically increasing the learning rate help?
A burst of higher rate can push the optimizer out of poor, sharp minima or off saddle points so it can find better regions.
What does the 'LR range test' do?
It runs briefly with a steadily rising rate; the loss curve reveals a sensible minimum and maximum to use for the cycles.
In the one-cycle policy, how does momentum typically behave relative to the learning rate?
The one-cycle policy lowers momentum while the rate peaks and raises it as the rate drops, which adds a regularizing effect.
What is 'snapshot ensembling' in the context of cyclical schedules?
Because each cycle settles into a different minimum, saving those checkpoints yields several diverse models from one run that can be ensembled.