Long Context vs RAG

Context windows grew from a few thousand tokens in early chat models to hundreds of thousands, and Google's Gemini 1.5 Pro, announced in 2024, offered a window of one million tokens. That raised an obvious question: if the model can read an entire manual, why build a retrieval pipeline at all? The answer depends on four trade-offs. Cost: providers charge per input token, so sending hundreds of thousands of tokens with every question costs far more than sending a few thousand relevant ones, although prompt caching can reduce the price of a repeated prefix. Latency: the model must process every input token before answering, so time to first token rises with context length. Accuracy: long context avoids retrieval misses, since nothing is filtered out, and it handles questions that need the whole document, like summarizing themes or comparing chapters. But models do not use all positions equally. The 2023 study Lost in the Middle found performance dropped when relevant information sat in the middle of long inputs, and needle-in-a-haystack tests measure only simple lookup, not reasoning across many scattered facts. Freshness and scale: RAG indexes can be updated incrementally, cover far more documents than any window, and support per-user access control by filtering what is retrieved. A practical rule: if the knowledge fits comfortably in the window, is queried repeatedly, and questions need broad understanding, long context is often simpler and better. If the corpus is large, changes often, has permissions, or must be answered cheaply with citations, retrieve. The common misconception is that one replaces the other. Many strong systems combine them: retrieve generously at the document level, then use a long window to include whole documents rather than small fragments.