Technical GUIDE
Cyclical Learning Rates
Cyclical learning rates repeatedly cycle the learning rate up and down between a lower and upper bound instead of only decaying it.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
This counterintuitive bouncing can speed up convergence and helps the optimizer escape sharp local minima and saddle points.
Deep Dive
Proposed by Leslie Smith in 2015, cyclical learning rates (CLR) challenge the assumption that the rate should only ever decrease. Instead, it oscillates between a minimum and maximum bound over a fixed number of iterations (a 'cycle'), often with a triangular shape. The intuition: periodically raising the rate provides a burst of energy that lets the model jump out of poor, sharp minima and traverse saddle points, while the low phases let it settle. Smith also introduced the 'LR range test' — a short run that sweeps the rate upward while watching the loss — to find good bounds automatically. Triangular, triangular-with-decay, and the famous one-cycle policy all build on this idea.
Real-World Implementation
fast.ai popularized the one-cycle policy as a default for quickly training image classifiers to high accuracy in few epochs.
The LR range test sweeps the rate upward over a few hundred batches to pick min and max bounds before a real run.
Snapshot ensembling saves a model checkpoint at the end of each cycle, producing a free ensemble from one training run.
Stochastic Gradient Descent with Warm Restarts (SGDR) periodically resets the rate to a high value to escape sharp minima.
The Future of Cyclical Learning Rates
Cyclical schedules and the one-cycle policy remain popular for fast training on vision and tabular tasks, and the LR range test is a standard tuning trick. For very large language models, smooth warmup-plus-cosine schedules tend to dominate, but the underlying insight — that strategic increases help escape bad regions of the loss landscape — informs warm restarts (SGDR) and ensemble methods that snapshot models at each cycle's low point. Expect continued cross-pollination between cyclical ideas and adaptive, self-tuning schedulers.
Putting it into practice
Why it matters
- Cost and budget. Architecture decisions drive performance and operating cost for years.
- Clearer decisions. Technical education helps teams choose the right stack, not just the newest one.
- Quality control. Better engineering choices reduce reliability incidents in production.
Watch out for
- Optimizing one benchmark can hide broader system weaknesses.
- Infrastructure and maintenance costs are often underestimated.
- Security and observability gaps can grow as systems become more complex.
Next steps
- Define latency, quality, and cost targets before implementation.
- Benchmark under realistic load and data conditions.
- Instrument monitoring for errors, drift, and user impact.
- Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Cyclical Learning Rates quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Cyclical Learning Rates?
Cyclical learning rates repeatedly cycle the learning rate up and down between a lower and upper bound instead of only decaying it. This counterintuitive bouncing can speed up convergence and helps the optimizer escape sharp local minima and saddle points.
What core assumption do cyclical learning rates challenge?
CLR deliberately raises the rate again and again, rejecting the traditional view that it must only decay monotonically.
What did Smith report for cyclical learning rates on tested classification tasks?
Smith reported improved classification accuracy without tuning and often in fewer iterations on the tested datasets; the paper does not guarantee a specific loss-landscape escape mechanism.
What does the 'LR range test' do?
It runs briefly with a steadily rising rate; the loss curve reveals a sensible minimum and maximum to use for the cycles.
In the one-cycle policy, how does momentum typically behave relative to the learning rate?
The one-cycle policy lowers momentum while the rate peaks and raises it as the rate drops, which adds a regularizing effect.
During a triangular cyclical learning-rate cycle, how does the rate move between its bounds?
A triangular policy increases the rate during one half-cycle and decreases it during the other half.
Keep learning
Related guides
More guides picked for this topic