KL Divergence Quiz

Answers and explanations appear as you take the quiz.

1. In D_KL(p||q), which distribution weights the log-ratio terms?
   • p
   • q
   • Their unweighted average
   • The distribution with larger entropy
2. Why is KL divergence not an ordinary symmetric distance?
   • It can only be negative
   • Swapping p and q generally changes the value
   • It has no logarithms
   • It is always exactly one
3. For p(x)>0 and q(x)=0, what happens to the corresponding forward KL contribution?
   • It is always one bit
   • It becomes zero
   • It diverges to infinity
   • It is omitted regardless of p
4. In the fair-versus-biased coin example, why can one summand be negative while total KL remains nonnegative?
   • The log base makes negative terms invalid
   • Every term is required to be positive
   • The formula takes an absolute value after summing
   • Individual log ratios can be negative, though the weighted total is nonnegative
5. What unit results from using base-two logarithms?
   • Nats
   • Bits
   • Meters
   • Percent points
6. How does cross-entropy relate to entropy and forward KL?
   • It is always zero when q differs
   • It equals D_KL(Q||P) minus entropy
   • H(P,Q) = H(P) + D_KL(P||Q)
   • It is the reciprocal of KL
7. In SciPy's entropy(pk, qk), what does supplying qk request?
   • A test of whether probabilities are independent
   • Only Shannon entropy of qk
   • A symmetric distance between both arrays
   • Relative entropy using pk relative to qk
8. Which statement about continuous KL is accurate?
   • It integrates a log density ratio under P and is infinite if P puts positive mass where Q has zero density
   • It sums point probabilities at every real-valued outcome
   • It is always finite whenever both distributions have densities
   • It has no support condition for the compared distributions