Ubuyobozi bwa tekiniki

KL Gutandukana

Kullback-Leibler (KL) divergence ipima igiciro cy'ubumenyi bwiyongereye bitezwe bwo gukoresha imwe mu mibare y'ibishoboka (probability distribution) kugira ngo isobanure indi.

Umva iki gitaboAI-byavuzwe · ~3 min

Kuriyi page3 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Kuzaza kwa KL Divergence
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Ni umubare utari mubi (nonnegative) ariko utandukanye (asymmetric), bityo guhindura imibare y'ibishoboka y'icyerekezo n'iyo ihuzanywe bisanzwe bihindura igisubizo kandi bishobora guhindura uko igitekerezo cy'uburyo bwo gushyiraho imibare giteye.

Kwibira cyane

For discrete distributions p and q over the same outcomes, KL divergence is D_KL(p||q) = sum over outcomes p(x) log(p(x)/q(x)). It compares q with a reference p, weighting each log ratio by how often the outcome occurs under p. In information theory, it can be interpreted as expected extra coding cost when a code is optimized for q but the true distribution is p. The log base sets the unit: natural logarithms yield nats and base two yields bits. KL divergence is nonnegative and equals zero when the distributions match on outcomes with positive probability under p, subject to support considerations. It is asymmetric: D_KL(p||q) is generally not D_KL(q||p). This makes it unsuitable as a conventional geometric distance. The direction matters in applications such as variational inference, where one distribution is treated as a target and another as an approximation. In the fair-versus-biased coin example, p=(0.5,0.5) and q=(0.9,0.1). The bit-valued contributions are 0.5 log2(0.5/0.9), about -0.424, and 0.5 log2(0.5/0.1), about 1.161. Their sum is about 0.737 bits. Negative individual terms are allowed even though the total divergence is nonnegative. Reversing p and q produces a different value because the weights also change. Support mismatches matter. If p assigns positive probability to an outcome where q assigns zero, the corresponding forward divergence is infinite. If both assign zero, that outcome contributes zero by convention in the discrete sum. For continuous distributions, KL is written as an integral of the log density ratio under P; finiteness requires P to be absolutely continuous with respect to Q. SciPy's entropy function computes Shannon entropy with one distribution and relative entropy when q is provided; check input normalization and direction. KL is useful for comparing distributions, but it does not by itself say whether a model is useful for a downstream decision.

Gushyira mu bikorwa Isi

Imibare y'ibishoboka y'icyuma gikomeye (fair coin) p=(0.5,0.5) ihuzanywe na q=(0.9,0.1). Mu bits, D_KL(p||q)=0.5 log2(0.5/0.9)+0.5 log2(0.5/0.1), hafi 0.737 bits.

Umuyoboro w'ururimi (language model) witeguriwe kugabanya KL kuva ku mibare y'ibishoboka y'icyerekezo ijya ku mibare y'ibishoboka y'ibivugwa. Niba amahirwe y'ibivugwa ku kintu cy'icyerekezo ari zeru, igiciro cya KL y'imbere gishobora kuba kinini cyane (infinite), bigasaba gusubiza neza cyangwa ubushobozi bw'umuyoboro.

Umushakashatsi ahuzanya D_KL(p||q) na D_KL(q||p) ku mibare ibiri y'ibishoboka akabona agaciro gatandukanye. Ibi bishobora kuba bitandukanye bivuze ko iyi mpuzandengo si intera isanzwe kandi nta gahigo rusange rya triangle inequality ifite.

Umushakashatsi akoresha scipy.stats.entropy(pk, qk) akemera ko ari rwo kintu gisobanura imibare y'ibishoboka y'icyerekezo kuko uburyo bwa API bwo guhuza imibare y'ibishoboka (relative-entropy direction) bukurikira pk ugereranyije na qk.

Kuzaza kwa KL Divergence

Ihuza ry'imibare y'ibishoboka rizasobanukirwa neza iyo raporo ivuga imibare y'ibishoboka y'icyerekezo, umubare w'ibanze wa log, n'uko ibintu bifite amahirwe ya zeru bikemurwa. Mu kwitegura imiyoboro, abashyirwa mu bikorwa bagomba guhuza icyerekezo cya KL n'uko bifuza ko ikintu gikora: gufata ibintu byose by'icyerekezo bitandukanye no gushyira imibare y'ibishoboka hafi y'icyerekezo cy'ingenzi. Gusesengura ibyemezo bikurikira ndetse n'agaciro ka divergence, kuko guhuza imibare y'ibishoboka bidahagije ntibihagije kugira ngo haboneke akamaro k'ukuri. Imiyoboro y'ejo hazaza ishobora kwinjiza gusuzuma ubushobozi no kureba uko gukora neza (smoothing) bihindura imibare, bigatuma ibibazo by'imibare by'impande z'inyuma biboneka aho guhindura amahirwe mu buryo butazwi.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Fata quiz ya KL Divergence

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

Ni iki KL Divergence?

Kullback-Leibler (KL) divergence ipima igiciro cy'ubumenyi bwiyongereye bitezwe bwo gukoresha imwe mu mibare y'ibishoboka kugira ngo isobanure indi. Ni umubare utari mubi (nonnegative) ariko utandukanye (asymmetric), bityo guhindura imibare y'ibishoboka y'icyerekezo n'iyo ihuzanywe bisanzwe bihindura igisubizo kandi bishobora guhindura uko igitekerezo cy'uburyo bwo gushyiraho imibare giteye.

Muri D_KL (p || q), ni ikihe gipimo gipima imvugo-igereranyo?

Igiteranyo kiremerewe na p (x), ikwirakwizwa ryerekanwa muri iki cyerekezo.

Kuki gutandukana kwa KL atari intera isanzwe?

Uburemere nigipimo gihinduka mugihe isaranganya ryunguranye.

Kuri p (x)> 0 na q (x) = 0, bigenda bite umusanzu wa KL ujyanye imbere?

Ikigereranyo cyibiti gifite imibare myiza na zeru zero, bitanga itandukaniro ritagira akagero.

Muburyo bwiza-bubogamye bwibiceri urugero, kuki umuntu ashobora guhamagarira kuba mubi mugihe KL yose ikomeza kuba idahwitse?

Bimwe q bishoboka birenze p, bitanga amagambo mabi, mugihe gutandukana kwuzuye byubahiriza nonnegativite.

Nibihe bice biva mugukoresha base-ebyiri logarithms?

Shingira ibipimo bibiri byamakuru muri bits; ibiti bisanzwe bitanga imbuto.