Jamba Hybrid Transformer-Ụdị Mamba
Jamba bụ nnukwu ụdị asụsụ sitere na AI21 Labs nke na-etinye nlebara anya Transformer n'ígwé mamba steeti-space layers (gbakwunyere ngwakọta-nke ndị ọkachamara) iji nweta arụmọrụ ogologo oge na-ahapụghị mma Transformer.
Nchịkọta
It matters because it shows hybrid architectures can beat pure Transformers on memory and throughput at long sequence lengths.
Ime miri emi
Ndị ngbanwe dị ọcha na-akwụ ụgwọ quadratic na nlebara anya ka ihe gbara ya gburugburu na-etolite, yana balloons cache uru bara uru nwere ogologo usoro. Ụdị oghere steeti dị ọcha dị ka nha Mamba n'ahịrị ma na-edobe ọnọdụ na-emegharị ugboro ugboro, mana akụkọ ihe mere eme anaghị etinye uche na ụfọdụ ọrụ. Jamba na-agwakọta ha abụọ: ọ na-akwakọba ihe mgbochi ebe ọtụtụ n'ígwé bụ Mamba (dị ọnụ ala, linear, dị ukwuu maka ogologo usoro) yana ọnụ ọgụgụ dị nta bụ nlebara anya ọkọlọtọ (siri ike na ncheta ziri ezi na echiche dị n'ime). Ọ na-agbakwụnye n'ígwé ngwakọta-nke ndị ọkachamara (MoE) iji tolite ikike ma na-edobe paramita na-arụ ọrụ nke ọma. Jamba mbụ wepụtara na windo ihe ngosi 256K-token ma nwee ike dabara nke ukwuu na otu GPU karịa ndị ntụgharị atụnyere, n'ihi cache KV ya pere mpe.
Nghọta nka nka
Mamba bụ ụdị oghere steeti ahọpụtara: kama ịga na akara ngosi ọ bụla gara aga, ọ na-edobe steeti abịakọrọ na-emegharị ugboro ugboro n'usoro n'usoro, yana ọnụ ụzọ ntinye dabere na-ekpebi ihe ị ga-edobe ma ọ bụ chefuo. Jamba na-etinye nlebara anya ole na ole nlebara anya n'etiti ọtụtụ ọkwa Mamba ka ihe nlereanya ahụ na-ejigide nlele anya ogologo oge ebe ọtụtụ n'ime mgbakọ na ebe nchekwa na-anọ n'ahịrị, usoro MoE na-arụ ọrụ naanị obere ndị ọkachamara n'otu akara.
Mmetụta atụmatụ
Ọsọ na ọnụ ọgụgụ
Usoro ọrụ asụsụ nwere ike ịga ngwa ngwa n'achụghị nkwụsi ike.
Nweta na iru
Ọ na-agbasawanye ohere n'ofe asụsụ na ụdị nzikọrịta ozi.
Mkpebi doro anya
Ndị otu nwere ike itinyekwu oge na ikpe ebe akpaaka na-ejikwa nkwughachi.
Ọdịnihu nke Jamba Hybrid Transformer-Mamba Model
Nlebara anya ngwakọ yana atụmatụ oghere steeti na-apụta dị ka ntụzịaka na-eduga maka ụdị ogologo okwu dị ogologo, Jamba nyere aka kwalite ụkpụrụ a. Na-atụ anya ka ndị ọzọ mepere emepe na nke dị n'okirikiri ka ị na-ewere nchịkọta agwakọta, megharịa nha anya-na-SSM, wee jikọta ha na aghụghọ MoE na KV-cache. Ka ihe ndị gbara ya gburugburu na-achọ ka ọ na-erute ọtụtụ nde akara ngosi, uru nchekwa-ahịrị-ncheta dị n'ígwé steeti-ohere na-eme ka ngwakọ mara mma karịsịa maka mbugharị ngwaọrụ yana ọnụ ahịa.
Mmejuputa n'ezie n'ụwa
Na-ahazi ntinye 256K-token dị ka ogologo ntinye akwụkwọ iwu ma ọ bụ nnukwu ebe nchekwa koodu na otu GPU nke na-enweghị ike dabara na cache Transformer's KV.
Ije ozi mkparịta ụka ogologo okwu dị elu ebe ọnọdụ Mamba na-eme ka ebe nchekwa dị larịị ka mkparịta ụka na-eto.
Nyocha akwụkwọ na iweghachi-ọgbọ agbakwunyere n'elu ntọala ihe ọmụma buru ibu etinyere ya ozugbo
Na-agba ọsọ ogologo okwu LLM nke mepere emepe (Jamba ka e wepụtara ya na nnukwu ibu mepere emepe) maka nyocha n'ime ụlọ ngwakọ.
Ihe ize ndụ & okporo ụzọ nche
Eziokwu ndị e chepụtara echepụta nwere ike jiri nwayọ tinye akụkọ, nkwado nkwado, ma ọ bụ nsonaazụ nyocha.
Mmetụta ngwa ngwa nwere ike ịmepụta nsonaazụ na-ekwekọghị ekwekọ n'ofe arịrịọ ndị yiri ya.
Enwere ike ikpughe data ederede nwere mmetụta ma ọ bụrụ na njikwa ohere adịghị ike.
Map mmejuputa
Kọwaa usoro mmepụta, ụda, na ụkpụrụ ịdịmma tupu ibugharị.
Weghachite nzaghachi site na isi mmalite ntụkwasị obi mgbe ọ bụla izi ezi dị mkpa.
Debe ebe nleba anya mmadụ maka mpụta dị elu.
Sochie ụkpụrụ ọdịda ma na-azụghachi mkpali ma ọ bụ usoro ọrụ mgbe niile.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Jamba Hybrid Transformer-Mamba Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Ụdị Oghere Ọchịchị steeti na Mamba
Ajụjụ a na-ajụkarị
What is Jamba Hybrid Transformer-Mamba Models?
Jamba bụ nnukwu ụdị asụsụ sitere na AI21 Labs nke na-etinye nlebara anya Transformer n'ígwé mamba steeti-space layers (gbakwunyere ngwakọta-nke ndị ọkachamara) iji nweta arụmọrụ ogologo oge na-ahapụghị mma Transformer. Ọ dị mkpa n'ihi na ọ na-egosi ngwakọ architectures nwere ike iti dị ọcha Transformers na ebe nchekwa na throughput na ogologo usoro ogologo.
Kedu ụdị isi oyi akwa abụọ Jamba na-ahapụ?
Njirimara Jamba na-akọwapụta na-achịkọta ọkwa oghere steeti Mamba yana ọnụ ọgụgụ dị nta nke nleba anya ntụgharị.
Kedu usoro ọzọ Jamba na-eji na-abawanye ikike ma na-edobe paramita na-arụ ọrụ dị ala?
Jamba na-agbakwụnye n'ígwé MoE ka ọ bụ naanị ndị ọkachamara na-arụ ọrụ n'otu akara, na-eto ike mkpokọta na-enweghị mkpokọta na-eto eto.
Kedu ihe kpatara Mamba ji dị mma karịa nlebara anya maka ogologo usoro?
Mamba na-edobe steeti mkpirisi, nke nwere ogologo emelitere n'ahịrị, na-ezere ọnụ ahịa nleba anya quadratic yana cache uru bara uru na-eto eto.
Kedu windo okirikiri ihe ngosi Jamba mbụ kwadoro?
Jamba jiri mpio ihe ngosi 256K-token butere nke ukwuu site na obere akara akara KV ya.
Kedu ihe kpatara Jamba ji edebe ụfọdụ nlebara anya karịa ịga Mamba dị ọcha?
Oghere nlebara anya zuru oke ole na ole na-edobe nleba anya nke ọma yana ike dị n'ime ebe ụdị oghere steeti dị ọcha nwere ike ịla azụ.