Penyematan Kalimat-BERT

A sentence embedding is a fixed-length list of numbers, a vector, that represents what a sentence means, so that sentences with similar meaning end up close together. Sentence-BERT, introduced by Nils Reimers and Iryna Gurevych of the UKP Lab at EMNLP 2019, is a way of producing such vectors from BERT, a widely used transformer language model. Before SBERT, BERT was very accurate at judging whether two sentences meant the same thing, but only when it read both sentences together. That design, now called a cross-encoder, is too slow for search, because every pair of sentences needs its own run through the network.

The SBERT paper puts the cost in concrete terms. Finding the most similar pair in a set of 10,000 sentences with a cross-encoder means about 50 million pairwise inferences, roughly 65 hours on an NVIDIA V100 GPU. With SBERT, each sentence is encoded once, about 5 seconds for all 10,000, and comparing the resulting vectors with cosine similarity, a measure of the angle between two vectors, takes about 0.01 seconds. The paper also shows why simply taking BERT's outputs does not work: averaging BERT's token outputs gave an average correlation with human similarity ratings of only 54.81 on seven semantic textual similarity (STS) tasks, and using the special [CLS] token gave 29.19, both worse than averaging older GloVe word vectors.

SBERT fixes this with fine-tuning in a siamese structure, meaning two copies of the same network with shared weights. Each sentence passes through BERT, and a pooling step turns the per-token outputs into one vector; the paper tested the [CLS] token, the mean of all token vectors and the maximum, and made mean pooling the default. The two sentence vectors, called u and v, are then trained with one of three objectives. For labeled pairs, a classifier sees u, v and their element-wise difference |u - v| and predicts a label. For similarity scores, the cosine similarity of u and v is trained to match a target score. For triplets, an anchor sentence is pulled toward a related sentence and pushed away from an unrelated one by at least a fixed margin.

The original models were trained on natural language inference (NLI) data, sentence pairs labeled as entailment, contradiction or neutral: about 570,000 pairs from SNLI and 430,000 from MultiNLI, with one epoch of training and a batch size of 16. On seven STS tasks, SBERT beat InferSent by 11.7 points and Universal Sentence Encoder by 5.5 points, and on the SentEval transfer benchmark it improved on them by 2.1 and 2.6 points. The trade-off showed up on the STS benchmark: a cross-encoding BERT-large fine-tuned on NLI and STS data scored 88.77, while the corresponding SBERT-large scored 86.10, a modest accuracy cost for a very large speed gain.

The idea grew into the Sentence Transformers library, created at UKP Lab and now maintained by Hugging Face. Its homepage lists more than 25,000 pretrained models on Hugging Face, plus support for cross-encoder rerankers, sparse encoders and multi-vector models. Later checkpoints moved from NLI to much larger contrastive training: the all-MiniLM-L6-v2 model card describes fine-tuning on about 1.17 billion sentence pairs from more than 20 datasets, including Reddit comments, S2ORC, WikiAnswers and MS MARCO, using a loss that scores every pair in a batch and rewards the true matches. In 2020 Reimers and Gurevych added a distillation method that extends an English model to other languages by training a student to map translations to the same vector as the original.

SBERT sits between two alternatives. Cross-encoders remain more accurate for scoring a single pair but cannot precompute anything, so they are used to rerank a short list. Older sentence encoders, such as averaged word vectors, InferSent and Universal Sentence Encoder, are cheaper but scored lower in the SBERT paper. Large embedding models offered through commercial APIs and newer open models now compete on the Massive Text Embedding Benchmark (MTEB), which in its 2022 paper covered 8 task types, 58 datasets and 112 languages across 33 models, and found that no single embedding method dominated every task.

The limitations are practical. Embeddings compress a whole passage into one vector, so long documents lose detail; all-MiniLM-L6-v2, for example, truncates input beyond 256 word pieces. Similarity in embedding space reflects the training data, so a model trained on general web pairs may misjudge legal, medical or code text. Because one vector summarizes a whole sentence, two sentences that differ only by a negation or a number can still land close together, so this is worth testing on real data. Embeddings from different models are not interchangeable, so switching models means re-encoding the whole collection. Bias in the training data carries into which texts count as similar.

The costs are modest compared with generative models. The main expenses are encoding a collection once and storing its vectors: one million 384-dimensional vectors in 32-bit floats take about 1.5 GB, and a larger 1,024-dimensional model would need about 4 GB for the same collection. Searching millions of vectors quickly calls for an approximate nearest-neighbor index, the approach the SBERT paper points to for reducing a 50-hour Quora search to milliseconds. Adapting a model to a specialized field means fine-tuning on pairs from that field with one of the objectives described above.