ELECTRA 사전 훈련

Masked language modeling trains on selected masked positions, whereas ELECTRA corrupts input tokens with plausible generator samples and trains a discriminator to classify each position as original or replaced. The generator is typically discarded after pretraining; the discriminator is fine-tuned for downstream tasks. The objective defines supervision over all input positions, which can improve sample efficiency under tested comparisons. The paper reports results for specified model sizes, data, and compute budgets, including strong performance for small models and competitive results at scale. These are experimental findings, not a guarantee that every ELECTRA model matches or beats BERT or uses a fixed fraction of compute. Generator cost, discriminator size, training recipe, and downstream benchmark all matter.