Il prossimoProssima guida
Apprendimento supervisionato
Fondamenti
GUIDA AI FONDAMENTALI
Self-supervised learning obtains a training signal from the data or transformations of it rather than requiring a task label for every example.
Ascolta questa guidaNarrato dall'IA · ~3 min
BERT and SimCLR illustrate different objectives for learning representations that can then be evaluated or adapted.
Self-supervised training constructs a prediction or comparison task from available data. This reduces the need for separate task annotations at that stage, while the data still requires collection, processing, and appropriate use. A derived training target is still supervision: self-supervised does not mean that learning has no objective or no target signal. BERT's masked-language-model objective selects token positions and predicts their original token identities using contextual information. The original procedure does not replace every selected position with a mask symbol. It can instead insert a random token or leave the token unchanged, while retaining the original token as the prediction target. That distinction separates the corrupted input from the training answer. Autoregressive language modeling uses another arrangement, predicting the next token from preceding text. SimCLR uses paired augmented views of an image. Two views from the same source image form a positive pair, and the contrastive objective compares their representations with other views under the defined setup. A learned nonlinear projection head transforms the encoder representation for that training objective. The original framework discards that projection head when using the encoder representation for downstream tasks. This is a particular architectural choice, not a required component of all self-supervised methods. Learned representations can be evaluated with a downstream classifier or adapted through fine-tuning, but transfer quality and label requirements depend on the task. Speech work such as wav2vec 2.0 provides another research example, with masked speech representations and subsequent supervised adaptation. Success on a pretraining objective does not itself establish reliable performance on every later application.
Studying BERT's selected-token prediction task before evaluating a language application.
Comparing representations from two augmented views in a SimCLR experiment.
Training an autoregressive model to predict the next token supplied by its text sequence.
Studying wav2vec 2.0 speech representations learned before supervised transcription adaptation.
Representation research can compare data transformations, prediction targets, and transfer evaluations across modalities. A pretraining method should be described through its actual objective and data rather than assumed to learn every desired property automatically. Downstream assessments can identify whether the representation works when frozen or needs adaptation and additional labels. Report the tested task and the amount of task annotation used. Broader or multimodal data can supply more learning opportunities, while data quality, resource cost, and the relevance of the objective remain practical factors to investigate.
Newsletter gratuita
Tre storie verificate sull'IA ogni mattina nei giorni feriali, scritte in un inglese semplice. Gratuito per sempre, senza pubblicità.
Un'email ogni giorno feriale. Disiscriviti con un clic. Non vendiamo né condividiamo mai il tuo indirizzo.
Mettiti alla prova
Feedback immediato su ogni risposta e un certificato condivisibile con un ID verificabile una volta superato un corso.
Sostieni l'istruzione gratuita sull'IA. AI Understanding è un'organizzazione non profit 501(c)(3) — nessun annuncio, nessun paywall, mai. Fai una donazione
Self-supervised learning obtains a training signal from the data or transformations of it rather than requiring a task label for every example. BERT and SimCLR illustrate different objectives for learning representations that can then be evaluated or adapted.
Un obiettivo o un confronto derivato fornisce comunque la supervisione, anche quando non è necessaria alcuna annotazione di attività separata per ciascun esempio di pre-addestramento.
L'obiettivo MLM originale prevede posizioni selezionate anziché ricostruire l'intero input; la corruzione applicata a tali posizioni può variare.
L'obiettivo pre-formazione trae il suo segnale dai dati; la raccolta, il calcolo e la successiva valutazione contano ancora.
L'uso a valle può includere la valutazione o la messa a punto delle funzionalità congelate, con prestazioni specifiche dell'attività e esigenze di etichetta valutate separatamente.
I modelli in stile GPT sono addestrati per prevedere il token successivo in una sequenza, mentre BERT prevede i token mascherati utilizzando sia il contesto sinistro che quello destro.
Continua a imparare
Altre guide selezionate per questo argomento
Il prossimoProssima guida
Apprendimento supervisionato
Fondamenti