Contextual Retrieval

Standard RAG splits documents into chunks, embeds each chunk, and retrieves the chunks most similar to a question. The problem is that splitting strips context. A chunk that says "the company's revenue grew by 3%" does not say which company or which period, so neither semantic search nor keyword search can reliably match it to a question that names them. Anthropic described contextual retrieval in September 2024. For each chunk, a language model is given the whole document and the chunk, and asked to write a brief context (typically around 50 to 100 tokens) that situates the chunk within the document. That context is prepended to the chunk. The combined text is then used in two indexes: an embedding index (contextual embeddings) and a BM25 keyword index (contextual BM25). At query time, results from both are combined, and optionally a reranker reorders the top candidates. In Anthropic's reported experiments across several datasets, contextual embeddings plus contextual BM25 reduced the top-20 retrieval failure rate by about 49 percent compared with standard embeddings, and adding reranking brought the reduction to about 67 percent. Results on your own data can differ, so it is worth measuring. The cost is at indexing time: one model call per chunk, each including the full document. Prompt caching makes this much cheaper, because the document is cached once and reused across all its chunks. Misconceptions to avoid: the added context is used for retrieval, and you can choose whether to show it to the generator. It does not replace good chunking, and for small knowledge bases that fit in the model's context window, sending everything directly may be simpler than any retrieval pipeline.