PANDUAN Teknikal

Divergens KL

Perbezaan Kullback-Leibler (KL) mengukur kos maklumat tambahan yang dijangkakan untuk menggunakan satu taburan kebarangkalian untuk mewakili yang lain.

Dengar panduan iniAI-diceritakan · ~3 min

Pada halaman ini3 min dibaca
  1. Gambaran keseluruhan
  2. Menyelam dalam
  3. Pelaksanaan Dunia Sebenar
  4. The Future of KL Divergence
  5. Mempraktikkannya
  6. Teruskan Meneroka
  7. Soalan lazim

Gambaran keseluruhan

It is nonnegative but asymmetric, so swapping the reference and comparison distributions generally changes the result and can change which modeling choice appears preferable.

Menyelam dalam

For discrete distributions p and q over the same outcomes, KL divergence is D_KL(p||q) = sum over outcomes p(x) log(p(x)/q(x)). It compares q with a reference p, weighting each log ratio by how often the outcome occurs under p. In information theory, it can be interpreted as expected extra coding cost when a code is optimized for q but the true distribution is p. The log base sets the unit: natural logarithms yield nats and base two yields bits. KL divergence is nonnegative and equals zero when the distributions match on outcomes with positive probability under p, subject to support considerations. It is asymmetric: D_KL(p||q) is generally not D_KL(q||p). This makes it unsuitable as a conventional geometric distance. The direction matters in applications such as variational inference, where one distribution is treated as a target and another as an approximation. In the fair-versus-biased coin example, p=(0.5,0.5) and q=(0.9,0.1). The bit-valued contributions are 0.5 log2(0.5/0.9), about -0.424, and 0.5 log2(0.5/0.1), about 1.161. Their sum is about 0.737 bits. Negative individual terms are allowed even though the total divergence is nonnegative. Reversing p and q produces a different value because the weights also change. Support mismatches matter. If p assigns positive probability to an outcome where q assigns zero, the corresponding forward divergence is infinite. If both assign zero, that outcome contributes zero by convention in the discrete sum. For continuous distributions, KL is written as an integral of the log density ratio under P; finiteness requires P to be absolutely continuous with respect to Q. SciPy's entropy function computes Shannon entropy with one distribution and relative entropy when q is provided; check input normalization and direction. KL is useful for comparing distributions, but it does not by itself say whether a model is useful for a downstream decision.

Pelaksanaan Dunia Sebenar

A hypothetical fair coin distribution p=(0.5,0.5) is compared with q=(0.9,0.1). In bits, D_KL(p||q)=0.5 log2(0.5/0.9)+0.5 log2(0.5/0.1), about 0.737 bits.

A language model is trained to minimize KL from a target token distribution to its predicted distribution. If the predicted probability for a target event is zero, the forward KL cost can become infinite, requiring careful smoothing or model support.

A researcher compares D_KL(p||q) with D_KL(q||p) for two distributions and finds different values. The asymmetry means the metric is not an ordinary distance and has no general triangle inequality.

An analyst uses scipy.stats.entropy(pk, qk) and confirms which argument represents the target distribution because the API's relative-entropy direction follows pk relative to qk.

The Future of KL Divergence

Distribution comparisons will be more interpretable when reports state which distribution is the reference, the log base, and how zero-probability events are handled. In model training, teams should connect KL direction to the behavior they want: covering all target modes differs from concentrating an approximation near a dominant mode. Evaluate downstream decisions as well as divergence values, since low distribution mismatch is not itself a guarantee of practical utility. Future pipelines can include support checks and sensitivity to smoothing, making numerical edge cases visible rather than silently altering probabilities.

Mempraktikkannya

Mengapa ia penting

  • Kos dan bajet. Keputusan seni bina memacu prestasi dan kos operasi selama bertahun-tahun.
  • Keputusan yang lebih jelas. Pendidikan teknikal membantu pasukan memilih timbunan yang betul, bukan hanya yang terbaharu.
  • Kawalan kualiti. Pilihan kejuruteraan yang lebih baik mengurangkan insiden kebolehpercayaan dalam pengeluaran.

Berhati-hati untuk

  • Mengoptimumkan satu penanda aras boleh menyembunyikan kelemahan sistem yang lebih luas.
  • Kos infrastruktur dan penyelenggaraan sering dipandang remeh.
  • Jurang keselamatan dan pemerhatian boleh berkembang apabila sistem menjadi lebih kompleks.

Langkah seterusnya

  1. Tentukan sasaran kependaman, kualiti dan kos sebelum pelaksanaan.
  2. Penanda aras di bawah beban realistik dan keadaan data.
  3. Pemantauan instrumen untuk ralat, drift dan kesan pengguna.
  4. Sediakan laluan balik dan tindak balas insiden sebelum penskalaan.

Teruskan Meneroka

Newsletter percuma

Dapatkan taklimat AI harian

Tiga cerita AI yang disahkan setiap pagi hari bekerja, ditulis dalam Bahasa Inggeris biasa. Percuma selamanya, tiada iklan.

Satu e-mel setiap hari bekerja. Berhenti melanggan dalam satu klik. Kami tidak pernah menjual atau berkongsi alamat anda.

Uji diri anda

Take the KL Divergence quiz

Maklum balas segera pada setiap jawapan, dan sijil yang boleh dikongsi dengan ID yang boleh disahkan sebaik sahaja anda lulus kursus.

Mulakan kuiz

Sokong pendidikan AI percuma. AI Understanding ialah pertubuhan bukan keuntungan 501(c)(3) — tiada iklan, tiada bayaran dinding, selamanya. Buat derma

Soalan lazim

What is KL Divergence?

Kullback-Leibler (KL) divergence measures the expected extra information cost of using one probability distribution to represent another. It is nonnegative but asymmetric, so swapping the reference and comparison distributions generally changes the result and can change which modeling choice appears preferable.

In D_KL(p||q), which distribution weights the log-ratio terms?

The sum is weighted by p(x), the reference distribution in this direction.

Why is KL divergence not an ordinary symmetric distance?

The weighting and ratio change when the distributions are exchanged.

For p(x)>0 and q(x)=0, what happens to the corresponding forward KL contribution?

The log ratio has a positive numerator and zero denominator, yielding infinite divergence.

In the fair-versus-biased coin example, why can one summand be negative while total KL remains nonnegative?

Some q probabilities exceed p, giving negative log terms, while the full divergence obeys nonnegativity.

What unit results from using base-two logarithms?

Base two measures information in bits; natural logs produce nats.