Multi-Head Latent Hankali
Multi-Head Latent Attention (MLA) wani tsarin kulawa ne, wanda aka gabatar a cikin DeepSeek-V2, wanda ke matsar da maɓalli-ƙimar maɓalli na yunwar ƙwaƙwalwar ajiya a cikin ƙaramin ɓoyayyen ɓoyayyen ɓoyayyen ɓoyayyiya.
Dubawa
It lets large language models run with far less GPU memory while keeping quality close to standard attention.
Zurfafa nutsewa
Lokacin da taswirar ta samar da rubutu, tana adana maɓalli da ƙima ga kowane alamar da ta gabata a cikin 'KV cache.' Wannan cache yana girma tare da tsawon mahallin kuma yana mamaye amfani da ƙwaƙwalwar ajiya yayin ƙaddamarwa. MLA tana maye gurbin manyan manyan maɓalli/ƙimar ƙima tare da ƙaramin madaidaicin madaidaicin sikelin kowane alama, sannan ayyukan da ke ɓoye baya zuwa maɓallan kowane kai da ƙima akan tashi. Saboda ƙananan latent ɗin kawai aka adana, DeepSeek-V2 ya ba da rahoton yanke ƙwaƙwalwar ajiyar KV-cache sama da 90% tare da daidaitaccen kulawar kai mai yawa, yana ba da damar yanayi mai tsayi da girma girma. Mahimmanci, za a iya naɗe matrices masu haɓakawa zuwa wasu ma'auni, don haka MLA ta cimma wannan matsawa tare da ƙaramin ko asara mai ƙima a cikin ingancin ƙirar ƙira.
Fahimtar Fasaha
MLA tana aiwatar da matsawar haɗin gwiwa mai ƙarancin daraja: kowane ɓoyayyen yanayin alama ana ƙididdige shi zuwa ƙarami mai ɓoyayyiyar ɓoyayyiyar ɓarna, da keɓance matrices masu tasowa suna sake gina maɓallan kowane kai da ƙima. Dabarar wayo ita ce 'ɗaukar' ma'aunin ƙima a cikin tambaya da tsinkayar fitarwa, don haka ƙirar ba ta taɓa samun cikakkun maɓalli/daraja yayin zayyana ba. Ana amfani da abubuwan sakawa na rotary tare da hanyar da aka yanke, tunda ba za'a iya juyar da jujjuya iri ɗaya ba, adana bayanan matsayi.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Mahimman Hankali na Shugabanni da yawa
MLA ta taimaka wajen samar da DeepSeek-V2 da V3 masu tattalin arziki don yin hidima a sikelin, kuma dabarar tana yaduwa yayin da ƙungiyoyi ke neman rahusa rahusa dogon yanayi. Yi tsammanin matsi na latent irin na MLA don haɗawa tare da yaduddukan Cakuɗa-na-Kwararru, ƙididdiga masu ƙididdigewa, da ƙididdige ƙididdiga a cikin buɗaɗɗen ƙira na gaba. Har ila yau, masu binciken suna nazarin yadda girman latent zai iya raguwa kafin ingancin ya ragu, kuma ko ra'ayin ƙananan matsayi na iya damfara hankali yayin horo, ba kawai tunani ba.
Aiwatar da Gaskiyar Duniya
Hidimar DeepSeek-V2/V3 ƙirar hira tare da ƙaramin sawun ƙwaƙwalwar GPU mai ban mamaki ga kowane buƙatu
Gudun dogon-takardun tambaya yana amsawa inda babban ma'ajin KV zai shayar da VRAM
Haɓaka girman batch ɗin ƙima akan ƙayyadaddun GPU saboda kowane jeri yana adana ɗan ƙaramin vector ne kawai
Bayar da dogon mahallin windows akan kayan masarufi don mataimakan da aka haɓaka
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Multi-Head Latent Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Hankalin Tambaya da yawa
Tambayoyin da ake yawan yi
What is Multi-Head Latent Attention?
Multi-Head Latent Attention (MLA) wani tsarin kulawa ne, wanda aka gabatar a cikin DeepSeek-V2, wanda ke matsar da maɓalli-ƙimar maɓalli na yunwar ƙwaƙwalwar ajiya a cikin ƙaramin ɓoyayyen ɓoyayyen ɓoyayyen ɓoyayyiya. Yana ba da damar manyan samfuran harshe suyi aiki tare da ƙarancin ƙwaƙwalwar GPU mai nisa yayin kiyaye inganci kusa da daidaitaccen kulawa.
Menene matsala ta farko da aka tsara Latent Latent Multi-Head don ragewa?
MLA tana hari da ma'ajin KV, wanda ke girma tare da tsawon mahallin kuma ya mamaye ƙwaƙwalwar ajiya yayin tsara rubutu.
Ta yaya MLA ke rage cache na KV?
MLA tana adana ƙaramin sikelin vector ɗaya a kowane alama kuma yana sake gina maɓalli da ƙima daga gare ta ta hanyar haɓakawa.
Wanne samfuri ne ya fara gabatar da Hankalin Latent Mai-Head?
DeepSeek ne ya gabatar da MLA a cikin ƙirar DeepSeek-V2 kuma an ɗauke shi zuwa DeepSeek-V3.
Me yasa MLA ke buƙatar keɓantaccen hanyar 'kyakkyawan' hanya don sakawa matsayi na juyawa?
Ba za a iya shigar da jujjuyawar jujjuyawar cikin wasu matrices masu nauyi ba, don haka MLA tana riƙe ƙaramin maɓalli mai mahimmanci don ɗaukar bayanan matsayi.
Kusan nawa KV-cache raguwar ƙwaƙwalwar ajiya ya yi rahoton DeepSeek-V2 daga MLA tare da daidaitaccen kulawar kai mai yawa?
DeepSeek-V2 ya ba da rahoton yanke ƙwaƙwalwar ajiyar KV-cache da fiye da 90%, yana ba da damar yanayi mai tsayi da manyan batches.