本頁閱讀時間 9 分鐘
概述
This late interaction keeps much of the accuracy of a full BERT reranker at a small fraction of the compute, which is why it is widely used for search and retrieval-augmented generation.
重點摘要
- ColBERT encodes queries and documents into one vector per token and scores them by summing each query token's maximum similarity to any document token.
- The 2020 SIGIR paper reports ColBERT reranking matched a BERT-base cross-encoder on MS MARCO while running over 170 times faster.
- ColBERTv2 compresses each token vector to 20 or 36 bytes with centroid-plus-residual quantization, shrinking the MS MARCO index from 154 GiB to 16 to 25 GiB.
- The PLAID engine cut ColBERTv2 search latency by up to 7 times on GPU and 45 times on CPU without reported quality loss.
- ColPali applies the same late interaction scoring to page images, letting systems retrieve PDFs and slides without extracting their text first.
深入探討
ColBERT, short for Contextualized Late Interaction over BERT, is a retrieval model: given a search query, it finds the most relevant passages in a large collection. Its distinctive idea is late interaction. Most neural search systems either compress each passage into a single vector and compare it with a single query vector, which is fast but loses detail, or feed the query and passage together through a transformer, which is accurate but far too slow to run against millions of passages. ColBERT sits between them. It encodes the query and each passage separately, so passage work can be done ahead of time, but it keeps one vector per token rather than one per passage, and only compares the two sets of vectors at the end, which is the late part of late interaction.
The mechanism has four steps. First, indexing: every passage is run through a BERT encoder, and each token's output is projected down to a 128-dimensional vector and normalized to unit length, with punctuation vectors dropped; these vectors are stored. Second, at query time, the query is encoded the same way, padded with special mask tokens to a fixed 32 positions so the model can add useful expansion terms. Third, scoring uses an operation called MaxSim: for each query vector, find the single passage vector with the highest similarity, then add up those maximum similarities across all query vectors. Fourth, candidates are found quickly by searching an approximate nearest-neighbor index for passage vectors close to each query vector, then only those candidate passages are scored exactly with MaxSim.
Omar Khattab and Matei Zaharia of Stanford introduced ColBERT in a paper posted to arXiv on April 27, 2020 and accepted at SIGIR 2020. On the MS MARCO passage ranking benchmark, measured by MRR@10, the average reciprocal rank of the first relevant result within the top ten, ColBERT reranking scored 34.9 against 34.7 for a BERT-base cross-encoder and 16.7 for the classic BM25 keyword method. It did so over 170 times faster, 61 milliseconds versus 10,700, with about 13,900 times fewer floating-point operations per query. Used end to end with a FAISS vector index, it reached 36.0. The cost was storage: indexing MS MARCO's roughly 9 million passages took about 3 hours on four GPUs and between 27 and 286 GiB depending on dimension and precision.
ColBERTv2, posted by Keshav Santhanam, Khattab and colleagues in December 2021 and published at NAACL 2022, fixed much of that storage problem and improved quality. It stores each token vector as the ID of its nearest cluster centroid plus a residual, the leftover difference, quantized to 1 or 2 bits per dimension, so a vector shrinks from 256 bytes to 20 or 36 bytes. The MS MARCO index fell from 154 GiB to 16 or 25 GiB, a 6 to 10 times reduction. It was also trained by distilling scores from a cross-encoder over hard negatives, reaching 39.7 MRR@10 on MS MARCO and the best result on 22 of 28 out-of-domain tests. PLAID, a May 2022 search engine for ColBERTv2, then cut search latency by up to 7 times on GPU and 45 times on CPU without reported quality loss, by first scoring passages using only their centroids.
Compared with the alternatives, ColBERT's trade-off is clear. Single-vector dense retrievers, the kind behind most embedding APIs, store one vector per passage and are cheaper to index and search, but squeeze all meaning into one point; the ColBERTv2 paper notes late interaction indexes were roughly an order of magnitude larger before compression. Sparse methods such as BM25 and learned sparse models like SPLADE use inverted indexes and match words exactly; in the ColBERTv2 evaluations, SPLADEv2 scored 36.8 on MS MARCO and still led on some tasks such as Climate-FEVER and HotpotQA. Cross-encoders, which read query and passage jointly, are the most precise but must run once per candidate at query time, so they are usually limited to reranking a short list, often one that ColBERT produced.
The idea has spread beyond text. ColPali, released in June 2024 and published at ICLR 2025, extends PaliGemma-3B, a vision-language model, to produce a set of 128-dimensional vectors for patches of a page image, then uses ColBERT-style MaxSim to match text queries against them. This lets systems retrieve PDFs, slides and scanned documents without a separate text-extraction step, and its authors introduced the ViDoRe benchmark to measure that kind of visually rich retrieval. The Hugging Face example for ColPali shows about 1,030 vectors for a single page, which illustrates why storage remains a central concern for multi-vector methods.
Limitations follow from the design. Storing a vector per token makes indexes larger than single-vector ones even after compression, and updating an index with new documents requires re-encoding those documents and sometimes re-clustering centroids. Query latency depends on a multi-stage pipeline with several tunable settings, such as how many centroid cells to probe and how many candidates to rescore, and poor settings trade away accuracy silently. Quality also depends on the training domain: ColBERTv2 was trained on MS MARCO web questions, and while it generalized well in its paper, it did not win every benchmark. Passage length matters too, since BERT-style encoders have a fixed maximum input, so long documents must be split into chunks before indexing.
In practice, running ColBERT requires a GPU for training and indexing, while search can run on CPU, according to the official repository. The Stanford code is MIT-licensed, ships a pretrained ColBERTv2 checkpoint trained on MS MARCO, and exposes a short Python workflow: an Indexer that builds a compressed index with a chosen number of bits, and a Searcher that returns the top k passages. The README reports indexing 10,000 passages in about six minutes on a free Colab T4 GPU. For most teams the decision comes down to whether the accuracy gain over a single-vector retriever on their own queries justifies a larger index and a more complex serving stack, which is best answered with a small evaluation set drawn from real traffic.
04工作範例
Scoring one passage with MaxSim by hand
Take a two-token query, 'solar panel', and a three-token passage, 'photovoltaic modules convert', each already encoded as unit vectors.
Compute the 2 by 3 grid of dot products; suppose 'solar' scores 0.62, 0.31 and 0.18 against the passage tokens, and 'panel' scores 0.40, 0.71 and 0.22.
For each query token, keep only its highest score: 0.62 for 'solar' and 0.71 for 'panel'.
Add the kept scores to get the passage score, 0.62 + 0.71 = 1.33.
Repeat for every candidate passage returned by the approximate nearest-neighbor stage.
Sort candidates by score and return the top k.
它顯示了什麼
The passage scores 1.33 even though it shares no exact words with the query, because each query token matched a semantically close passage token. Check the result by confirming the score never exceeds the number of query tokens, since each cosine maximum is at most 1.
現實世界的實施
Reranking: the 2020 ColBERT paper reranked BM25's top 1,000 MS MARCO passages at 34.9 MRR@10 in 61 milliseconds, against 34.7 for a BERT-base cross-encoder that took 10,700 milliseconds.
End-to-end retrieval over large collections with Stanford's open-source ColBERTv2 and PLAID code, which the PLAID paper evaluated on collections of up to 140 million passages.
Retrieval-augmented generation pipelines built with RAGatouille or DSPy, both listed in the official ColBERT repository as ways to plug ColBERTv2 into applications that feed passages to a language model.
ColPali, a PaliGemma-3B extension published in 2024, applies late interaction to page images, retrieving scanned PDFs and slides by their visual content instead of extracted text.
Out-of-domain search over niche forums, measured by the LoTTE benchmark of StackExchange questions, where ColBERTv2 led on all topics in its 2021 paper.
The Future of ColBERT and Late Interaction Retrieval
The main open problem for late interaction is cost per document. ColBERTv2's residual compression (2021) and PLAID's centroid pruning (2022) brought storage and latency into practical range for text, but multi-vector representations of images, as in ColPali (2024), produce around a thousand vectors per page, which revives the same storage and speed questions at larger scale. Research directions visible in the literature include tighter compression, pruning which token vectors are worth keeping, and better engines for multi-vector search in general-purpose vector databases. A second thread is generalization. ColBERTv2 performed strongly out of domain in its own evaluations, while learned sparse models led on some tasks, and benchmarks such as LoTTE (2021) and ViDoRe (2024) were created specifically because standard sets did not capture long-tail or visually rich retrieval. As retrieval-augmented generation grows, the direction of travel is toward evaluating retrievers on end-task quality rather than ranking metrics alone. Whether late interaction, single-vector, sparse or hybrid retrieval wins for a given application remains an empirical question that depends on the corpus, the queries and the latency budget.
付諸實踐
為什麼這很重要
- 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
- 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
- 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。
留意
- 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
- 及時的敏感性可能會在類似的請求中產生不一致的結果。
- 如果存取控制薄弱,敏感文字資料可能會暴露。
後續步驟
- 在推出之前定義輸出格式、語氣和品質標準。
- 當準確性很重要時,請使用可信任來源進行地面回應。
- 為高風險輸出保留人工審查檢查點。
- 追蹤故障模式並定期重新訓練提示或工作流程。
資料來源與延伸閱讀
- arXiv / SIGIR 2020 (Khattab, Zaharia)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
- arXiv / NAACL 2022 (Santhanam et al.)ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
- arXiv (Santhanam et al.)PLAID: An Efficient Engine for Late Interaction Retrieval
- arXiv / ICLR 2025 (Faysse et al.)ColPali: Efficient Document Retrieval with Vision Language Models
- Hugging Facevidore/colpali model card
- Stanford FutureData (GitHub)stanford-futuredata/ColBERT
不斷探索
常見問題
What is ColBERT and Late Interaction Retrieval?
ColBERT is a neural search model from Stanford that encodes a query and each document as one vector per token, then scores a match by letting every query token find its most similar document token and adding those scores. This late interaction keeps much of the accuracy of a full BERT reranker at a small fraction of the compute, which is why it is widely used for search and retrieval-augmented generation.
ColBERT 如何表示查詢或文件?
ColBERT 為每個標記保留單獨的上下文嵌入,而不是將所有內容折疊到一個向量中。
ColBERT 使用的評分函數叫什麼?
MaxSim 取得每個查詢標記相對於所有文件標記的最大相似度,並對這些最大值求和。
What query-time work does ColBERT avoid by precomputing document token vectors?
Independent document encoding moves substantial work offline; it does not eliminate encoding the new query or scoring candidates.
「遲到的互動」中的「遲到」指的是什麼?
首先對查詢和文件進行單獨編碼;他們的細微互動發生在 MaxSim 評分過程中的後期。
ColBERT 解決了單向量密集檢索的什麼問題?
Representing an entire passage as one vector can lose token-level detail; ColBERT retains separate contextual token vectors. Single-vector encoders need not compute a simple average.