Cache-Augmented Generation (CAG)

Cache-augmented generation, or CAG, replaces the retrieval step with preloading. Instead of searching a knowledge base for each question, the system feeds the entire knowledge base to the model once, saves the resulting key-value (KV) cache, and reuses that cache for every subsequent query. The approach was described in a December 2024 paper titled Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks. The KV cache is the set of intermediate attention keys and values a transformer computes for every token it has read. Normally it is built during a single request and discarded. In CAG, the documents are processed ahead of time, the cache is stored, and each question is appended after the cached prefix. The model only computes the new question tokens before generating, so answers start faster than if the documents were re-read each time. After answering, the appended tokens are truncated so the cache returns to its documents-only state for the next query. The benefits are no retrieval errors, since every document is always present, no embedding index or vector database to maintain, and lower per-query latency than naive long context. The limits follow directly: the whole corpus must fit in the model's context window, the model must still reason well over that much text, and any change to the documents means rebuilding the cache. CAG therefore fits small, stable knowledge bases such as a product manual, a policy set, a syllabus or a fixed regulatory text. It is a poor fit for large, fast-changing or permission-filtered collections. A common misconception is that CAG trains the model. Nothing is fine-tuned; the weights are unchanged. Commercial prompt caching from API providers is a hosted relative of the same idea.