Ikibazo gihuza ibibazo · Hagati Urwego

Ikigororotse-Binyuze mu Ikigereranyo

Ikizamini gikomeye imbere yindangagaciro, surrogate gradients, kubogama, gukoresha umubare, gukoresha logique, gushyira mubikorwa isuzuma, hamwe no gukata buhoro buhoro.

Inzira zijyanye no kuyoboraUgororotse Binyuze mu Kugereranya
Ikibazo 1 Bya 8

How does the straight-through estimator handle a hard operation?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. How does the straight-through estimator handle a hard operation?
    • Replace the forward result with a probability vector
    • Stop all gradients before the operation
    • Use the hard result forward and approximate the backward gradient with an identity or smooth proxy
    • Compute an exact derivative for every discontinuity
  2. Why can rounding block ordinary gradient descent?
    • Its derivative is zero almost everywhere and undefined at jumps
    • Rounding always increases loss
    • The input becomes categorical
    • It removes the training labels
  3. What is the bias of a straight-through gradient estimate?
    • It always equals the exact gradient
    • It is unbiased for every hard function
    • It is a random label rather than a gradient
    • It uses a surrogate gradient that differs from the true derivative
  4. Which example can use an STE?
    • A continuous linear layer only
    • A quantization step that rounds activations during the forward pass
    • A dataset split with no model computation
    • A metric computed after training
  5. What does `x + stop_gradient(hard(x) - x)` achieve conceptually?
    • A hard forward value with gradients that behave like the identity on x
    • A smooth forward value with no gradient
    • A random category sampled from x
    • A hard value with exact derivative of the rounding function
  6. Why might a clipped pass-through gradient be used?
    • To make the forward value continuous
    • To turn the estimator unbiased
    • To limit gradient flow outside a chosen range for stability
    • To replace the model’s activation function
  7. Does an STE compute the true derivative of a hard threshold?
    • Yes, because thresholding is smooth
    • No; it substitutes a practical gradient approximation
    • Yes, if the model is quantized
    • It computes no forward value
  8. What is a sensible evaluation after training with an STE?
    • Evaluate only the surrogate gradient
    • Assume the estimator proves accuracy
    • Remove the hard operation at inference without testing
    • Measure task performance using the actual hard quantized or thresholded operation