BERT- und Encoder-Modelle

The 2018 BERT paper describes Bidirectional Encoder Representations from Transformers. Its encoder attention uses context from both sides of an allowed input position, unlike a purely left-to-right causal attention pattern. This is an architectural distinction rather than proof of perfect language understanding. Original BERT pretraining combines masked language modeling with next-sentence prediction. The masked objective selects token positions for prediction; the input at those positions can be masked, randomly replaced, or unchanged under the stated original procedure. It is inaccurate to say that all selected positions are necessarily hidden by the same mask symbol. Next-sentence prediction constructs sentence pairs with labels indicating whether the second sentence is the actual continuation or a randomly sampled alternative. These targets are derived from the corpus and do not require a human relationship label for every pair. BERT input embeddings combine token, position, and segment embeddings. Segment embeddings identify the two portions in an original sentence-pair input; position embeddings supply location information. After pretraining, task-specific heads can use the contextual outputs for classification or span prediction, and fine-tuning adapts parameters to the task. This does not guarantee a small adaptation cost or accuracy improvement in every setup. Later variants use different mechanisms. DistilBERT applies distillation for a smaller student, while ALBERT reduces parameter requirements through factorized embeddings and cross-layer sharing. Fewer stored parameters should not be treated as universal speed or accuracy superiority. Sentence-BERT further adapts pretrained encoders to yield sentence representations that can be compared independently for similarity tasks.