Negative Sampling and Noise Contrastive Estimation Quiz

Answers and explanations appear as you take the quiz.

1. What problem do negative sampling and NCE primarily solve?
   • The high cost of computing a full softmax over a very large output space
   • Overfitting on small datasets
   • Vanishing gradients in recurrent networks
   • The lack of labeled data for supervised learning
2. How does Noise Contrastive Estimation reframe the learning task?
   • As regression on continuous targets
   • As binary classification between true data samples and noise samples
   • As clustering of unlabeled points
   • As reinforcement learning with rewards
3. Which model popularized negative sampling for learning word embeddings?
   • BERT
   • ResNet
   • word2vec's skip-gram model
   • GPT-2
4. How does negative sampling differ from full NCE?
   • It adds extra normalization terms for accuracy
   • It requires the entire vocabulary at every step
   • It uses a regression loss instead of a sigmoid
   • It drops NCE's normalization terms, making it faster but not a consistent density estimator
5. Why are negatives often sampled from the unigram distribution raised to the 3/4 power?
   • To balance the influence of frequent and rare words rather than sampling purely by frequency
   • To ignore rare words entirely
   • To make all words equally likely
   • To remove stop words from training
6. For a large vocabulary, how can sampled negative pairs reduce work compared with evaluating every word in a full softmax?
   • It computes an exact probability for every vocabulary item
   • The update scores a positive pair and a limited set of sampled negatives instead of normalizing over the whole vocabulary
   • It eliminates the need to score the positive pair
   • It sorts all words by frequency on each step
7. In skip-gram negative sampling, what is treated as the positive training pair?
   • Two words sampled independently from the noise distribution
   • A word paired with an empty context
   • A word and a context word observed together in the corpus
   • A validation label paired with a model checkpoint
8. Why is negative sampling not generally interpreted as estimating a normalized language-model probability?
   • It always sums over the full vocabulary
   • It requires exact maximum-likelihood normalization
   • It never uses negative examples
   • Its simplified binary objective drops NCE normalization terms and targets useful embeddings instead