Ururimi AI

Amagambo-BERT

Sentence-BERT (SBERT) is a method, published at EMNLP 2019, that fine-tunes BERT-style models so each sentence becomes a single vector whose distance to other vectors reflects meaning.

Umva iki gitaboAI-byavuzwe · ~8 min

Kuriyi page9 min soma
  1. Incamake
  2. Ibyingenzi byingenzi
  3. Kwibira cyane
  4. Building a tiny semantic search with Sentence Transformers
  5. Gushyira mu bikorwa Isi
  6. Ejo hazaza h'interuro-BERT
  7. Kubishyira mubikorwa
  8. Inkomoko no gusoma
  9. Komeza Ubushakashatsi
  10. Ibibazo bikunze kubazwa

Incamake

It cut the time to find the most similar pair among 10,000 sentences from about 65 hours to about 5 seconds, and its open-source library underpins much of today's semantic search and retrieval.

Ibyingenzi byingenzi

  1. Sentence-BERT fine-tunes BERT in a siamese setup so each sentence gets one vector that can be compared with cosine similarity.
  2. The 2019 paper cut the search for the most similar pair among 10,000 sentences from about 65 hours to about 5 seconds.
  3. Averaging raw BERT outputs or using the CLS token produced worse sentence similarity scores than averaging GloVe word vectors.
  4. Cross-encoders are slightly more accurate for scoring pairs, so many systems retrieve with embeddings and rerank with a cross-encoder.
  5. The Sentence Transformers library lists more than 25,000 pretrained models on Hugging Face and is maintained by Hugging Face.

Kwibira cyane

A sentence embedding is a fixed-length list of numbers, a vector, that represents what a sentence means, so that sentences with similar meaning end up close together. Sentence-BERT, introduced by Nils Reimers and Iryna Gurevych of the UKP Lab at EMNLP 2019, is a way of producing such vectors from BERT, a widely used transformer language model. Before SBERT, BERT was very accurate at judging whether two sentences meant the same thing, but only when it read both sentences together. That design, now called a cross-encoder, is too slow for search, because every pair of sentences needs its own run through the network.

The SBERT paper puts the cost in concrete terms. Finding the most similar pair in a set of 10,000 sentences with a cross-encoder means about 50 million pairwise inferences, roughly 65 hours on an NVIDIA V100 GPU. With SBERT, each sentence is encoded once, about 5 seconds for all 10,000, and comparing the resulting vectors with cosine similarity, a measure of the angle between two vectors, takes about 0.01 seconds. The paper also shows why simply taking BERT's outputs does not work: averaging BERT's token outputs gave an average correlation with human similarity ratings of only 54.81 on seven semantic textual similarity (STS) tasks, and using the special [CLS] token gave 29.19, both worse than averaging older GloVe word vectors.

SBERT fixes this with fine-tuning in a siamese structure, meaning two copies of the same network with shared weights. Each sentence passes through BERT, and a pooling step turns the per-token outputs into one vector; the paper tested the [CLS] token, the mean of all token vectors and the maximum, and made mean pooling the default. The two sentence vectors, called u and v, are then trained with one of three objectives. For labeled pairs, a classifier sees u, v and their element-wise difference |u - v| and predicts a label. For similarity scores, the cosine similarity of u and v is trained to match a target score. For triplets, an anchor sentence is pulled toward a related sentence and pushed away from an unrelated one by at least a fixed margin.

The original models were trained on natural language inference (NLI) data, sentence pairs labeled as entailment, contradiction or neutral: about 570,000 pairs from SNLI and 430,000 from MultiNLI, with one epoch of training and a batch size of 16. On seven STS tasks, SBERT beat InferSent by 11.7 points and Universal Sentence Encoder by 5.5 points, and on the SentEval transfer benchmark it improved on them by 2.1 and 2.6 points. The trade-off showed up on the STS benchmark: a cross-encoding BERT-large fine-tuned on NLI and STS data scored 88.77, while the corresponding SBERT-large scored 86.10, a modest accuracy cost for a very large speed gain.

The idea grew into the Sentence Transformers library, created at UKP Lab and now maintained by Hugging Face. Its homepage lists more than 25,000 pretrained models on Hugging Face, plus support for cross-encoder rerankers, sparse encoders and multi-vector models. Later checkpoints moved from NLI to much larger contrastive training: the all-MiniLM-L6-v2 model card describes fine-tuning on about 1.17 billion sentence pairs from more than 20 datasets, including Reddit comments, S2ORC, WikiAnswers and MS MARCO, using a loss that scores every pair in a batch and rewards the true matches. In 2020 Reimers and Gurevych added a distillation method that extends an English model to other languages by training a student to map translations to the same vector as the original.

SBERT sits between two alternatives. Cross-encoders remain more accurate for scoring a single pair but cannot precompute anything, so they are used to rerank a short list. Older sentence encoders, such as averaged word vectors, InferSent and Universal Sentence Encoder, are cheaper but scored lower in the SBERT paper. Large embedding models offered through commercial APIs and newer open models now compete on the Massive Text Embedding Benchmark (MTEB), which in its 2022 paper covered 8 task types, 58 datasets and 112 languages across 33 models, and found that no single embedding method dominated every task.

The limitations are practical. Embeddings compress a whole passage into one vector, so long documents lose detail; all-MiniLM-L6-v2, for example, truncates input beyond 256 word pieces. Similarity in embedding space reflects the training data, so a model trained on general web pairs may misjudge legal, medical or code text. Because one vector summarizes a whole sentence, two sentences that differ only by a negation or a number can still land close together, so this is worth testing on real data. Embeddings from different models are not interchangeable, so switching models means re-encoding the whole collection. Bias in the training data carries into which texts count as similar.

The costs are modest compared with generative models. The main expenses are encoding a collection once and storing its vectors: one million 384-dimensional vectors in 32-bit floats take about 1.5 GB, and a larger 1,024-dimensional model would need about 4 GB for the same collection. Searching millions of vectors quickly calls for an approximate nearest-neighbor index, the approach the SBERT paper points to for reducing a 50-hour Quora search to milliseconds. Adapting a model to a specialized field means fine-tuning on pairs from that field with one of the objectives described above.

04Urugero rwakazi

Building a tiny semantic search with Sentence Transformers

  1. Install the library with pip install sentence-transformers.

  2. Load a small model: model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2').

  3. Write five short documents, for example sentences about cooking, weather, sports, finance and travel, and encode them with model.encode(docs).

  4. Encode a query that shares no keywords with its best match, such as 'Will it rain tomorrow?' when a document says 'The forecast calls for showers.'

  5. Compute scores with model.similarity(query_embedding, doc_embeddings) and sort the documents by score.

  6. Add a negated sentence, such as 'The forecast calls for no showers', and see how close its score is to the original.

Ibyo byerekana

The learner gets a ranked list in which the weather sentence scores highest despite sharing no words with the query. The negation test shows how far apart the model places two sentences with opposite meanings; if the scores are close, that is a direct demonstration of why many systems add a cross-encoder reranker or other checks.

Gushyira mu bikorwa Isi

Semantic search: the all-MiniLM-L6-v2 model from the Sentence Transformers project maps text to 384-dimensional vectors for retrieval and clustering, and its Hugging Face model card shows about 192 million downloads in a single month.

Duplicate question detection: the SBERT paper notes that comparing a new question against Quora's more than 40 million questions with cross-encoding BERT would take over 50 hours, while precomputed embeddings plus an index make it a matter of milliseconds.

Retrieval-augmented generation pipelines embed document chunks once with a Sentence Transformers model, store the vectors, and fetch the closest chunks for each user question before a language model writes an answer.

Multilingual search: Reimers and Gurevych's 2020 knowledge-distillation method trains a student model so that translated sentences land near the original sentence's vector, with results reported for more than 50 languages.

Two-stage ranking: the Sentence Transformers library pairs fast embedding models with Cross-Encoder rerankers, which score a query and a document together to reorder the top results from a vector search.

Ejo hazaza h'interuro-BERT

Sentence embeddings have shifted from a research topic to infrastructure for search and retrieval-augmented generation, and most of the open questions now concern quality across domains and languages. The MTEB paper (2022) found that no single embedding method won across all of its tasks, which suggests that choosing a model for a specific domain still requires testing on that domain's data. The Sentence Transformers library has expanded beyond single dense vectors. Its documentation now covers cross-encoder rerankers, sparse encoders that produce keyword-like weights, and multi-vector encoders for late-interaction retrieval in the style of ColBERT, reflecting a move toward combining several retrieval signals instead of relying on one vector per passage. Open problems include representing long documents without losing detail, handling negation and fine-grained factual differences, keeping embeddings compatible when models are updated, and extending quality to languages with little training data. The 2020 multilingual distillation work showed one route, transferring an English model's vector space through translations, and its authors reported results for more than 50 languages. How well such methods hold up for low-resource languages and specialized fields remains an active research question.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Umuvuduko n'igipimo. Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
  • Kugera no kugera. Yagura uburyo bwindimi nuburyo bwo gutumanaho.
  • Ibyemezo bisobanutse. Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.

Witondere

  • Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
  • Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
  • Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.

Intambwe ikurikira

  1. Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
  2. Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
  3. Komeza kugenzura abantu kugenzura ibisubizo byinshi.
  4. Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.

Inkomoko no gusoma

  1. arXiv (EMNLP 2019)Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
  2. Sentence TransformersSentenceTransformers Documentation
  3. Hugging Facesentence-transformers/all-MiniLM-L6-v2
  4. arXivMTEB: Massive Text Embedding Benchmark
  5. arXiv (EMNLP 2020)Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Sentence-BERT Embeddings quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Sentence-BERT Embeddings?

Sentence-BERT (SBERT) is a method, published at EMNLP 2019, that fine-tunes BERT-style models so each sentence becomes a single vector whose distance to other vectors reflects meaning. It cut the time to find the most similar pair among 10,000 sentences from about 65 hours to about 5 seconds, and its open-source library underpins much of today's semantic search and retrieval.

Ni ikihe kibazo cyibanze hamwe na BERT isanzwe interuro-BERT ikemura?

Ikibaya cya BERT kigomba gutunganya interuro ebyiri, bityo rero ingano nini yo kugereranya ntishobora kubarwa; SBERT ikora buri nteruro inshuro imwe muri vector ikoreshwa.

Ni ubuhe buryo bwubaka urusobe Sentence-BERT ikoresha?

SBERT ikoresha imiterere ya siamese (impanga) aho kodegisi ebyiri za BERT zisangira uburemere kandi buri kimwe kigashyiramo interuro imwe yigenga.

Which pooling strategy did the original Sentence-BERT paper evaluate for creating sentence embeddings?

The original paper compares pooling strategies, including mean pooling over token embeddings. This is a paper-specific experimental result, not a claim that one strategy is universally most recommended.

Iyo interuro zimaze gushyirwamo, ni gute ubusanzwe busa?

Ingingo yose ya SBERT nuko guhuza bigabanuka kuri cosine ihendutse / akadomo-gicuruzwa ugereranije hagati ya vectors.

What training data did the original Sentence-BERT paper use for NLI fine-tuning?

The paper fine-tunes SBERT on Natural Language Inference data, which includes entailment, contradiction and neutral examples. This states the paper’s training setup rather than a universal dataset preference.