Nleba anya nleba anya nke isi ọtụtụ
Multi-Head Latent Attention (MLA) bụ usoro nlebara anya, ewepụtara na DeepSeek-V2, nke na-akpakọba nchekwa uru igodo-agụụ na-agụ n'ime obere vector latent nkekọrịta.
Nchịkọta
It lets large language models run with far less GPU memory while keeping quality close to standard attention.
Ime miri emi
Mgbe ihe ngbanwe na-ewepụta ederede, ọ na-echekwa igodo na uru vector maka akara ngosi ọ bụla gara aga na 'cache KV'. Cache ahụ na-eto n'ogologo okirikiri ma na-achịkwa iji ebe nchekwa n'oge ntinye. Onye omebe iwu na-eji otu akara vector dị ala dị ala dochie ọtụtụ igodo/uru vector zuru oke, wee rụọ ọrụ na-eweghachite n'ime igodo otu isi na ụkpụrụ na ofufe. N'ihi na ọ bụ naanị kọmpat kọmpat ka echekwara, DeepSeek-V2 kọrọ na-ebelata ebe nchekwa KV-cache karịa 90% na nlebara anya ọtụtụ isi, na-eme ka ọnọdụ dị ogologo na nha batch buru ibu. N'ụzọ dị oke mkpa, enwere ike mpịaji matrices elu-elu n'ime nha ndị ọzọ, yabụ onye omebe iwu na-enweta mkpakọ a na obere mfu ma ọ bụ enweghị atụ na ịdịmma ụdị.
Nghọta nka nka
MLA na-eme mkpakọ nkwonkwo dị ala: ọnọdụ ezoro ezo nke ọ bụla ka a na-atụpụta ya ka ọ bụrụ obere vector latent, wee kewapụ matrices elu-elu na-ewughachi igodo otu isi na ụkpụrụ. Aghụghọ dị nkọ bụ 'ịnabata' ibu ọrụ elu n'ime ajụjụ na ntule mmepụta, yabụ na ihe nlereanya ahụ anaghị emepụta igodo/ụkpụrụ zuru oke n'oge ntinye. A na-eji ụzọ igodo agbawapụrụ agbaji eme ihe ntinye ọnọdụ rotary, ebe ọ bụ na enweghị ike itinye uche n'otu ụzọ ahụ, na-echekwa ozi ọnọdụ.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke Multi-Head Latent Ntị
MLA nyere aka mee DeepSeek-V2 na V3 ka ọ bụrụ akụ na ụba iji jee ozi n'ọtụtụ, usoro a na-agbasa ka ndị otu na-achụso ntinye ogologo okwu dị ọnụ ala karịa. Na-atụ anya mkpakọ latent n'ụdị MLA ka ọ ga-ejikọta ya na akwa ngwakọta-nke ndị ọkachamara, cache agụpụtara, na nhọpụta nkọwa n'ụdị mepere emepe n'ọdịnihu. Ndị na-eme nchọpụta na-enyochakwa ókè akụkụ latent nwere ike ịdalata tupu ịdị mma adalata, yana ma otu echiche dị ala nwere ike ịkpakọba uche n'oge ọzụzụ, ọ bụghị naanị ntinye.
Mmejuputa n'ezie n'ụwa
Na-eje ozi ụdị nkata DeepSeek-V2/V3 nwere akara ukwu ebe nchekwa GPU pere mpe nke ukwuu maka arịrịọ ọ bụla
Na-agba ọsọ ajụjụ akwụkwọ ogologo na-aza ebe nnukwu oghere KV ga-agwụcha VRAM
Na-abawanye nha ntinye ntinye na GPU edoziri n'ihi na usoro nke ọ bụla na-echekwa naanị obere vector latent
Na-eme ka windo ndị dị ogologo dị ogologo na ngwa ngwa ngwa ahịa maka ndị enyemaka eweghachiri eweghachi
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Multi-Head Latent Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Nleba anya ajụjụ ọtụtụ
Ajụjụ a na-ajụkarị
What is Multi-Head Latent Attention?
Multi-Head Latent Attention (MLA) bụ usoro nlebara anya, ewepụtara na DeepSeek-V2, nke na-akpakọba nchekwa uru igodo-agụụ na-agụ n'ime obere vector latent nkekọrịta. Ọ na-ahapụ ụdị asụsụ buru ibu ka ha nwee ebe nchekwa GPU dị obere ka ha na-edobe ịdịmma nso na nlebara anya ọkọlọtọ.
Gịnị bụ isi nsogbu Multi-Head Latent Attention e mere iji belata?
Onye omebe iwu lekwasịrị anya cache KV, nke na-eto n'ogologo okirikiri ma na-achịkwa ebe nchekwa n'oge ọgbọ ederede.
Kedu ka MLA si emebi cache KV?
MLA na-echekwa otu vector latent kọmpat n'otu akara wee wughachi igodo na ụkpụrụ sitere na ya site na nrụpụta elu.
Kedu ụdị mbụ webatara nleba anya nleba anya nke ọtụtụ isi?
DeepSeek webatara MLA na ụdị DeepSeek-V2 wee buru ya na DeepSeek-V3.
Kedu ihe kpatara onye omebe iwu ji chọọ ụzọ 'decouped' dị iche maka ntinye ọnọdụ rotary?
Enweghị ike itinye uche na mgbanwe rotary n'ime matrices dị arọ ndị ọzọ, yabụ onye omebe iwu na-edobe obere ihe agbakwunyere agbakwunyere iji buru ozi ọnọdụ.
Kedu ihe dị ka mbelata ebe nchekwa KV-cache ka DeepSeek-V2 kọrọ site na MLA na nlebara anya ọtụtụ isi?
DeepSeek-V2 kọrọ na-ebelata ebe nchekwa KV-cache site na ihe karịrị 90%, na-eme ka ọnọdụ dị ogologo na nnukwu batches.