Ọmụma Distillation
Ọmụma distillation na-azụ obere 'ụmụ akwụkwọ' nlereanya iji ṅomie nnukwu ihe nlereanya 'onye nkuzi' ziri ezi.
Nchịkọta
It matters because it shrinks powerful models so they run cheaply on phones and servers while keeping much of the accuracy.
Ime miri emi
Modelsdị buru ibu bụ nke ziri ezi mana ọ dị nwayọ ma dịkwa ọnụ ibu ibu. Ntugharị ihe ọmụma na-ebufe ike ha ka ọ bụrụ kọmpat nlereanya site n'ime ka nwa akwụkwọ ahụ mụta ihe site na ihe onye nkụzi na-eme ka ọ bụghị nanị site na akara ike. Nghọta bụ isi, site na Hinton na ndị ọrụ ibe, bụ na nkesa nke puru omume nke onye nkuzi na-ebu 'ihe ọmụma gbara ọchịchịrị': ọbụlagodi mgbe ọ na-ebu amụma 'nkịta', ihe gbasara nke puru omume maka 'wolf' megide 'ụgbọ ala' na-ekpughe otú onye nkụzi si ahụ myirịta. Iji okpomọkụ mee ka ihe puru omume ndị a dị nro na-ekpughe ihe owuwu ahụ, a zụrụ nwa akwụkwọ ahụ ka ọ dakọtara ya, na-esonyekarị na ezigbo akara. Ihe si na ya pụta bụ obere ihe nlere ngwa ngwa nke na-achịkọta nke ọma karịa nke a zụrụ na akara naanị. DistilBERT na TinyBERT bụ ụdị asụsụ agbasasị ama ama.
Nghọta nka nka
Ọnwụ nke kpochapụwo na-ejikọta okwu nhụsianya (KL divergence n'etiti ihe omume dị nro nke nwa akwụkwọ na onye nkụzi) yana ọkọlọtọ cross-entropy na ezigbo akara. Ịdị nro na-eji okpomọkụ T na softmax: elu T na-eme ka nkesa dị ntakịrị ka obere myirịta dị n'etiti klas ghọọ akara mmụta; A na-ejikarị T-squared na-agbazi gradient ahụ. Ọdịiche dị iche iche na-agabiga nsonaazụ: njiri njiri arụ ọrụ dabara n'ọkwa zoro ezo dị n'etiti, yana nhụsianya dabere na mmekọrịta dabara na mmekọrịta n'etiti ọmụmaatụ.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Ntugharị Ọmụma
Distillation bụ usoro ọkọlọtọ ugbu a na mbupu ụdị dị mma yana ọ bụ isi na ebili mmiri nke obere obere ụdị mepere emepe taa. Usoro na-eto ngwa ngwa bụ nhụsianya n'usoro n'usoro site na ụdị asụsụ buru ibu, ebe ihe nlereanya siri ike na-emepụta data ọzụzụ ma ọ bụ usoro ntụgharị uche (gụnyere usoro echiche) iji kuziere ụmụ akwụkwọ nta ihe, na-eji data sịntetịt na-eme ka ahịrị ghara ịdị. Na-atụ anya ijikọ nke ọma na quantization na ịkwachaa, ibunye ngwaọrụ ọzọ, yana arụmụka na-aga n'ihu gbasara inye ikike na ịdị mma mgbe ị na-atụgharị site na ụdị nwe ya nke nsonaazụ ya na-aghọ akara ọzụzụ ndị asọmpi.
Mmejuputa n'ezie n'ụwa
DistilBERT na-akpakọ BERT ka ọ bụrụ 40% pere mpe ka ọ na-ejigide ọtụtụ nghọta asụsụ ya maka ntinye ngwa ngwa.
Na-ebelata nnukwu ụdị ọhụụ ka ihe nhazi ihe onyonyo wee na-agba ọsọ ozugbo na ngwa igwefoto ama ama.
Na-atụgharị uche n'echiche nke nnukwu ihe nlereanya n'ime obere ihe nlereanya iji mee ka ọ zaa mgbakọ na mwepụ ma ọ bụ itinye ajụjụ ọnụ karịa ọnụ ala.
Na-akpakọkọta mkpokọta ụdị n'ime otu nwa akwụkwọ ka mmepụta ihe na-akwụ ụgwọ na-adalata na-enweghị nnukwu mfu.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Knowledge Distillation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Njikwa Ọmụma AI
Ajụjụ a na-ajụkarị
What is Knowledge Distillation?
Ọmụma distillation na-azụ obere 'ụmụ akwụkwọ' nlereanya iji ṅomie nnukwu ihe nlereanya 'onye nkuzi' ziri ezi. Ọ dị mkpa n'ihi na ọ na-ebelata ụdị dị ike nke mere na ha na-agba ọsọ dị ọnụ ala na ekwentị na sava ebe ha na-edebe ọtụtụ izi ezi.
In knowledge distillation, what is the student model trained to do?
Distillation na-ebufe ikike onye nkuzi ka ọ bụrụ kọmpat, ngwa ngwa ihe nlereanya ụmụ akwụkwọ.
Gịnị ka 'ihe ọmụma gbara ọchịchịrị' nke onye nkụzi pụtara?
Ihe ọmụma gbara ọchịchịrị bụ usoro dị na nkesa nke puru omume nke onye nkuzi, dịka otu 'wolf' si dị na 'nkịta', ọ bụghị naanị azịza kachasị elu.
Kedu ọrụ okpomọkụ (T) na-arụ na distillation?
Okpomọkụ dị elu na-eme ka nkesa nke puru omume, na-ekpughe myirịta ndị nwa akwụkwọ kwesịrị ịmụta.
The kpochapụwo distillation ọnwụ na-agwakọta nke abụọ components?
Nwa akwụkwọ a dabara na nkesa nro dị nro nke onye nkuzi (KL) ebe ọ dabara na akara nke eziokwu ala (cross-entropy).
Kedu ihe atụ nke ụdị asụsụ distilled?
DistilBERT bụ ihe atụ ama ama nke esitere na BERT, na-ejigide ọtụtụ arụmọrụ yana oke pere mpe.