Technical GUIDE

Warmup and Cosine Annealing Schedules

Warmup gently ramps the learning rate up from near zero before training, then cosine annealing smoothly decays it back down following a cosine curve.

2 min read
Last updated Oct 2, 2026
By AI Understanding Editorial

Listen to this guideAI-narrated · ~2 min

Start reading Take the quiz

Technical · 2 min read

Where you'll see it

1GPT-style and BERT-style language models use a linear warmup over the first ~1-2% of steps followed by cosine decay to near zero.
2Vision transformers (ViT) train with cosine annealing and a short warmup to avoid early divergence on ImageNet.
3Hugging Face Transformers offers `get_cosine_schedule_with_warmup` as a one-line scheduler for fine-tuning jobs.

Learning RateTransformerGradient

Check yourselfQuiz available

On this page2 min read

Overview
Deep Dive
Real-World Implementation
The Future of Warmup and Cosine Annealing Schedules
Putting it into practice
Keep Exploring
Frequently asked questions

Overview

Together they stabilize early training and squeeze out better final accuracy, which is why nearly every modern transformer is trained this way.

Deep Dive

When training starts, model weights are random and gradients can be huge, so jumping straight to a large learning rate often causes loss spikes or divergence — especially with adaptive optimizers like Adam, whose variance estimates are unreliable in the first steps. Warmup fixes this by linearly increasing the rate over a few hundred to a few thousand steps. Once the model is on stable footing, cosine annealing takes over, decaying the rate as 0.5 * (1 + cos(pi * t / T)) of its peak. The cosine shape keeps the rate high early for fast progress, then eases off gradually so the optimizer can settle into a good minimum instead of bouncing around it.

Real-World Implementation

GPT-style and BERT-style language models use a linear warmup over the first ~1-2% of steps followed by cosine decay to near zero.

Vision transformers (ViT) train with cosine annealing and a short warmup to avoid early divergence on ImageNet.

Hugging Face Transformers offers `get_cosine_schedule_with_warmup` as a one-line scheduler for fine-tuning jobs.

Stable Diffusion and other diffusion models fine-tune with warmup to prevent gradient explosions when adapting pretrained weights.

The Future of Warmup and Cosine Annealing Schedules

Warmup-plus-cosine remains the default recipe for large language models, but variants are spreading. Warmup-stable-decay (WSD) keeps a constant rate then decays sharply at the end, making it easy to extend runs without recommitting to a fixed length. Researchers are also studying why warmup works — linking it to gradient noise and loss-landscape curvature — and tools increasingly auto-tune warmup length and peak rate, reducing the manual trial-and-error that dominates today.

Putting it into practice

Why it matters

Cost and budget. Architecture decisions drive performance and operating cost for years.
Clearer decisions. Technical education helps teams choose the right stack, not just the newest one.
Quality control. Better engineering choices reduce reliability incidents in production.

Watch out for

Optimizing one benchmark can hide broader system weaknesses.
Infrastructure and maintenance costs are often underestimated.
Security and observability gaps can grow as systems become more complex.

Next steps

Define latency, quality, and cost targets before implementation.
Benchmark under realistic load and data conditions.
Instrument monitoring for errors, drift, and user impact.
Prepare rollback and incident response paths before scaling.

Keep Exploring

AI Benchmarks

Use evaluation properly when comparing technical options.Read Guide

Reinforcement Learning

Go deeper into technical training strategies.Read Guide

Free newsletter

Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.

One email each weekday. Unsubscribe in one click. We never sell or share your address.

Test yourself

Take the Warmup and Cosine Annealing Schedules quiz

Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.

Start quiz

Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation

Frequently asked questions

What is Warmup and Cosine Annealing Schedules?

Warmup gently ramps the learning rate up from near zero before training, then cosine annealing smoothly decays it back down following a cosine curve. Together they stabilize early training and squeeze out better final accuracy, which is why nearly every modern transformer is trained this way.

What is the main purpose of the warmup phase at the start of training?

Starting at a large learning rate on random weights can cause loss spikes or divergence, so warmup ramps the rate up gently to keep early steps stable.

Cosine annealing decays the learning rate following which shape?

The rate is scaled by 0.5 * (1 + cos(pi * t / T)), producing a smooth hill that stays high early and glides to near zero at the end.

Which optimizer especially benefits from warmup because its variance estimates are unreliable early on?

Adam's running variance estimates are based on very few samples in the first steps, making the effective step size erratic — warmup mitigates this.

In the cosine formula, what does T represent?

T is the horizon over which the rate decays from its peak to near zero; t is the current step within that horizon.

In the guide’s warmup-plus-cosine schedule, what begins after warmup reaches its target rate?

The guide describes a short linear warmup followed by cosine annealing.

Keep learning