Text Embeddings

Text representations vary in the unit they encode and the objective used to learn them. A static word embedding assigns a stored vector to a vocabulary entry. This makes a lookup for the same entry unchanged across sentences, even when a word such as bank has different senses. Contextual encoders instead compute token representations using the surrounding input. BERT studies a bidirectional transformer encoder, but producing context-dependent states is not the same as guaranteeing useful sentence-similarity scores. Sentence-BERT addresses sentence representation through a modified training and inference setup. The original paper adds pooling to BERT or RoBERTa outputs, experiments with several pooling strategies, and uses mean pooling as its default. Its siamese architecture has tied encoder weights, allowing sentences to be encoded independently and compared afterward. This differs from a cross-encoder that processes a pair jointly to obtain a pair-specific score. The distinction matters when many stored candidates must be compared with new text, because independent document vectors can be computed ahead of a query. Similarity must still use the method appropriate to the trained model. Sentence Transformers documents cosine, dot product, and negative distance alternatives; cosine is its documented default, not the only valid metric for all text embeddings. Dense Passage Retrieval trains question and passage encoders and ranks passages with their vector dot product. This is one retrieval design that can supply context to an answer system. Retrieval-augmented generation is not defined by requiring a particular embedding model, and a highly ranked passage does not by itself guarantee a correct final answer. Evaluate both the retrieved evidence and the generated response.