概述
Long context is often simpler and better when the knowledge fits the window and questions need a whole-document view; RAG usually wins on cost per query, latency, freshness, scale and access control. Most strong systems combine the two rather than choosing one.
深入探討
Context windows grew from a few thousand tokens in early chat models to hundreds of thousands, and Google's Gemini 1.5 Pro, announced in 2024, offered a window of one million tokens. That raised an obvious question: if the model can read an entire manual, why build a retrieval pipeline at all? The answer depends on four trade-offs. Cost: providers charge per input token, so sending hundreds of thousands of tokens with every question costs far more than sending a few thousand relevant ones, although prompt caching can reduce the price of a repeated prefix. Latency: the model must process every input token before answering, so time to first token rises with context length. Accuracy: long context avoids retrieval misses, since nothing is filtered out, and it handles questions that need the whole document, like summarizing themes or comparing chapters. But models do not use all positions equally. The 2023 study Lost in the Middle found performance dropped when relevant information sat in the middle of long inputs, and needle-in-a-haystack tests measure only simple lookup, not reasoning across many scattered facts. Freshness and scale: RAG indexes can be updated incrementally, cover far more documents than any window, and support per-user access control by filtering what is retrieved. A practical rule: if the knowledge fits comfortably in the window, is queried repeatedly, and questions need broad understanding, long context is often simpler and better. If the corpus is large, changes often, has permissions, or must be answered cheaply with citations, retrieve. The common misconception is that one replaces the other. Many strong systems combine them: retrieve generously at the document level, then use a long window to include whole documents rather than small fragments.
現實世界的實施
A lawyer loads one 300-page merger agreement into a long-context model to ask how indemnity terms interact across sections, a question that scattered retrieved chunks would answer poorly.
A customer support platform with millions of help articles that change daily uses RAG, because no context window holds the corpus and the index can be updated article by article.
A company chatbot serving employees with different clearance levels uses retrieval with permission filters, so each user's prompt only contains documents they are allowed to see.
A research team retrieves the ten most relevant papers, then passes each full paper into a long context window instead of small fragments, combining both approaches.
The Future of Long Context vs RAG
Larger windows and caching discounts move the break-even point toward long context for small and medium corpora. Retrieval is unlikely to disappear, because many organizations hold far more text than any window, their data changes constantly, and they need per-user permissions and citations. The more likely direction is blending: agents that decide when to retrieve, read whole documents once found, and cache what they reuse. Research continues on making models use middle positions reliably and on cheaper attention for long inputs. Because prices and models change often, teams should rerun their own cost and accuracy comparisons periodically.
付諸實踐
為什麼這很重要
- 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
- 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
- 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。
留意
- 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
- 及時的敏感性可能會在類似的請求中產生不一致的結果。
- 如果存取控制薄弱,敏感文字資料可能會暴露。
後續步驟
- 在推出之前定義輸出格式、語氣和品質標準。
- 當準確性很重要時,請使用可信任來源進行地面回應。
- 為高風險輸出保留人工審查檢查點。
- 追蹤故障模式並定期重新訓練提示或工作流程。
不斷探索
常見問題
What is Long Context vs RAG?
Long context means placing entire documents directly into a model's large context window, while RAG retrieves only the most relevant passages for each question. Long context is often simpler and better when the knowledge fits the window and questions need a whole-document view; RAG usually wins on cost per query, latency, freshness, scale and access control. Most strong systems combine the two rather than choosing one.
為什麼長上下文方法的每次查詢成本通常比 RAG 更高?
成本會隨輸入標記而變化,因此每次發送整個語料庫的成本比發送一些相關段落的成本要高得多。
2023 年「迷失在中間」研究發現了什麼?
模型在長輸入的開始和結束處使用的資訊比中間的資訊更可靠。
大海撈針測試的主要限制是什麼?
找到一個植入的事實比組合透過文件傳播的多個事實更容易。
與長上下文相比,哪種情況最明顯有利於 RAG?
RAG 可以擴展到任何視窗之外,增量更新,並且可以根據權限過濾結果。
當使用長語料庫的提示快取時,提示應該如何安排?
快取重複使用相同的前綴,因此不變的內容必須先出現。