ColBERT and Late Interaction Retrieval

ColBERT, short for Contextualized Late Interaction over BERT, is a retrieval model: given a search query, it finds the most relevant passages in a large collection. Its distinctive idea is late interaction. Most neural search systems either compress each passage into a single vector and compare it with a single query vector, which is fast but loses detail, or feed the query and passage together through a transformer, which is accurate but far too slow to run against millions of passages. ColBERT sits between them. It encodes the query and each passage separately, so passage work can be done ahead of time, but it keeps one vector per token rather than one per passage, and only compares the two sets of vectors at the end, which is the late part of late interaction.

The mechanism has four steps. First, indexing: every passage is run through a BERT encoder, and each token's output is projected down to a 128-dimensional vector and normalized to unit length, with punctuation vectors dropped; these vectors are stored. Second, at query time, the query is encoded the same way, padded with special mask tokens to a fixed 32 positions so the model can add useful expansion terms. Third, scoring uses an operation called MaxSim: for each query vector, find the single passage vector with the highest similarity, then add up those maximum similarities across all query vectors. Fourth, candidates are found quickly by searching an approximate nearest-neighbor index for passage vectors close to each query vector, then only those candidate passages are scored exactly with MaxSim.

Omar Khattab and Matei Zaharia of Stanford introduced ColBERT in a paper posted to arXiv on April 27, 2020 and accepted at SIGIR 2020. On the MS MARCO passage ranking benchmark, measured by MRR@10, the average reciprocal rank of the first relevant result within the top ten, ColBERT reranking scored 34.9 against 34.7 for a BERT-base cross-encoder and 16.7 for the classic BM25 keyword method. It did so over 170 times faster, 61 milliseconds versus 10,700, with about 13,900 times fewer floating-point operations per query. Used end to end with a FAISS vector index, it reached 36.0. The cost was storage: indexing MS MARCO's roughly 9 million passages took about 3 hours on four GPUs and between 27 and 286 GiB depending on dimension and precision.

ColBERTv2, posted by Keshav Santhanam, Khattab and colleagues in December 2021 and published at NAACL 2022, fixed much of that storage problem and improved quality. It stores each token vector as the ID of its nearest cluster centroid plus a residual, the leftover difference, quantized to 1 or 2 bits per dimension, so a vector shrinks from 256 bytes to 20 or 36 bytes. The MS MARCO index fell from 154 GiB to 16 or 25 GiB, a 6 to 10 times reduction. It was also trained by distilling scores from a cross-encoder over hard negatives, reaching 39.7 MRR@10 on MS MARCO and the best result on 22 of 28 out-of-domain tests. PLAID, a May 2022 search engine for ColBERTv2, then cut search latency by up to 7 times on GPU and 45 times on CPU without reported quality loss, by first scoring passages using only their centroids.

Compared with the alternatives, ColBERT's trade-off is clear. Single-vector dense retrievers, the kind behind most embedding APIs, store one vector per passage and are cheaper to index and search, but squeeze all meaning into one point; the ColBERTv2 paper notes late interaction indexes were roughly an order of magnitude larger before compression. Sparse methods such as BM25 and learned sparse models like SPLADE use inverted indexes and match words exactly; in the ColBERTv2 evaluations, SPLADEv2 scored 36.8 on MS MARCO and still led on some tasks such as Climate-FEVER and HotpotQA. Cross-encoders, which read query and passage jointly, are the most precise but must run once per candidate at query time, so they are usually limited to reranking a short list, often one that ColBERT produced.

The idea has spread beyond text. ColPali, released in June 2024 and published at ICLR 2025, extends PaliGemma-3B, a vision-language model, to produce a set of 128-dimensional vectors for patches of a page image, then uses ColBERT-style MaxSim to match text queries against them. This lets systems retrieve PDFs, slides and scanned documents without a separate text-extraction step, and its authors introduced the ViDoRe benchmark to measure that kind of visually rich retrieval. The Hugging Face example for ColPali shows about 1,030 vectors for a single page, which illustrates why storage remains a central concern for multi-vector methods.

Limitations follow from the design. Storing a vector per token makes indexes larger than single-vector ones even after compression, and updating an index with new documents requires re-encoding those documents and sometimes re-clustering centroids. Query latency depends on a multi-stage pipeline with several tunable settings, such as how many centroid cells to probe and how many candidates to rescore, and poor settings trade away accuracy silently. Quality also depends on the training domain: ColBERTv2 was trained on MS MARCO web questions, and while it generalized well in its paper, it did not win every benchmark. Passage length matters too, since BERT-style encoders have a fixed maximum input, so long documents must be split into chunks before indexing.

In practice, running ColBERT requires a GPU for training and indexing, while search can run on CPU, according to the official repository. The Stanford code is MIT-licensed, ships a pretrained ColBERTv2 checkpoint trained on MS MARCO, and exposes a short Python workflow: an Indexer that builds a compressed index with a chosen number of bits, and a Searcher that returns the top k passages. The README reports indexing 10,000 passages in about six minutes on a free Colab T4 GPU. For most teams the decision comes down to whether the accuracy gain over a single-vector retriever on their own queries justifies a larger index and a more complex serving stack, which is best answered with a small evaluation set drawn from real traffic.