概述
The choice matters because the first mainly shifts what a model knows about a domain and its language, the second mainly shifts how it responds, and the data and compute needed differ greatly.
深入探討
Continued pretraining, also called domain-adaptive pretraining, picks up where the original pretraining stopped. The model sees raw text from the target domain, such as medical papers, source code or legal filings, and learns to predict every next token, exactly as in pretraining. Gururangan and colleagues showed in the 2020 paper Don't Stop Pretraining that a second phase of pretraining on in-domain text improved RoBERTa on tasks in biomedical, computer science, news and review domains. It usually needs large volumes of text, often billions of tokens, and full-parameter training, so it costs far more than fine-tuning. Fine-tuning, usually supervised fine-tuning, uses prompt and response pairs. Loss is typically computed only on the response tokens, so the model learns what to produce given an input. Datasets range from a few thousand to hundreds of thousands of examples, and parameter-efficient methods such as LoRA make it affordable on modest hardware. It is good at teaching format, style, task behavior and instruction following. Preference methods such as RLHF or DPO often follow. A common pipeline is base model, then continued pretraining, then supervised fine-tuning, then preference tuning. Continued pretraining has a known risk: catastrophic forgetting, where general skills degrade. Applied to a chat model, it can also erode instruction following, so teams often start from a base model and redo instruction tuning afterward. Two misconceptions stand out. First, fine-tuning is not a reliable way to add many new facts; research such as Gekhman and colleagues (2024) found models learn new facts slowly through fine-tuning and that doing so can increase hallucination. For fast-changing facts, retrieval is often the better tool. Second, BloombergGPT is sometimes cited as continued pretraining, but it was trained from scratch on a mix of financial and general data. A rough rule: use continued pretraining when domain language differs a lot from web text and unlabeled data is plentiful; use fine-tuning when you need a behavior or format.
戰略影響
成本與預算
多年來，架構決策決定著效能和營運成本。
更明確的決策
技術教育幫助團隊選擇正確的堆疊，而不僅僅是最新的堆疊。
品質管控
更好的工程選擇可以減少生產中的可靠性事故。
The Future of Continued Pretraining vs Fine-Tuning
The steady release of capable open base models has made continued pretraining practical for more organizations, though it still requires substantial data and compute. Active research areas include reducing forgetting, making domain adaptation cheaper, and combining weights from separately trained models. In practice, many teams will keep pairing light fine-tuning with retrieval rather than full continued pretraining, reserving the heavier option for domains whose language is poorly covered by general web data.
現實世界的實施
Code Llama was built by continuing to train Llama 2 on a large corpus of code, and some of its variants were then further trained for Python and for following instructions.
A legal-tech company with millions of unlabeled contracts and filings runs continued pretraining to improve the model's handling of legal language, then fine-tunes on a few thousand annotated clause-extraction examples.
A bank wants a chatbot that answers in a fixed JSON format with a set tone; supervised fine-tuning, or even careful prompting, is enough because the model already knows the general content.
EPFL's Meditron models adapted Llama 2 to medicine through continued pretraining on medical papers and clinical guidelines before any task-specific fine-tuning.
風險與防護欄
優化一項基準測試可以隱藏更廣泛的系統弱點。
基礎設施和維護成本常常被低估。
隨著系統變得更加複雜，安全性和可觀察性差距可能會擴大。
實施路線圖
在實施之前定義延遲、品質和成本目標。
在實際負載和資料條件下進行基準測試。
儀器監控錯誤、漂移和使用者影響。
在擴展之前準備回滾和事件回應路徑。
不斷探索
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Continued Pretraining vs Fine-Tuning quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
常見問題
What is Continued Pretraining vs Fine-Tuning?
Continued pretraining keeps training a base model with the same next-token objective on large amounts of raw domain text, while fine-tuning trains on a smaller curated set of labeled examples, such as instruction and response pairs, to shape behavior. The choice matters because the first mainly shifts what a model knows about a domain and its language, the second mainly shifts how it responds, and the data and compute needed differ greatly.
What is the main data difference between continued pretraining and supervised fine-tuning?
Continued pretraining predicts next tokens on raw text, while fine-tuning learns from prompt and response pairs.
In supervised fine-tuning, where is the loss usually computed?
Masking the prompt means the model learns what to produce given an input, not to reproduce the input.
What is replay in continued pretraining?
Including general data helps keep general skills from degrading while the model learns the domain.
How was Code Llama built?
Code Llama is a continued pretraining example, with some variants further trained for Python and instructions.
For facts that change often, which approach does the guide suggest is often better than fine-tuning?
Fine-tuning learns new facts slowly and can increase hallucination, so retrieval is often better for changing information.
繼續學習
相關指南
為此主題精選的更多指南