Jamba Hybride Transformateur-Mamba
Jamba xeetu làkk bu mag la bu bawoo ci AI21 Labs buy boole ay couche yuy bàyyi xel ci Transformer ak ay couche espace-state Mamba (ak njaxasu eksper yi) ngir am njariñu contexte bu yàgg te baña bàyyi kalite Transformer bi.
Résumé
Dafa am solo ndax dafay wane ni architecture hybrid yi mën nañu raw Transformers yu sell yi ci memory ak ci produit yi ci guddaayi sekans yu gudd.
Plongeur bu xóot
Transformers yu sell yi dañuy fay njëgu quadratic ci bàyyi xel ci ni contexte bi di màggee, ak seen balloon cache bu am valeur bu am solo ak guddaayu toppalante bi. Modèlu espace-state bu sell bu melni Mamba dafay scale lineairement ba noppi tëye stade recurrent bu am dayo fixe, waaye ci taarix dañuy bàyyi xel ci yenn liggéey yi. Jamba dafay boole ñaari mbir yooyu: dafay jël ay blok fu layer yu bari nekk Mamba (yomb, linear, baax ci sequences yu gudd) ak limu gëna ndaw ñuy bàyyi xel ci standard (am doole ci fàttaliku bu leer ak xalaat ci contexte). Dafay yokk itam ay njaxasu-ekspert (MoE) ngir yokk kàttan gi, boole ci tëye ay parametre yuy dox. Jamba bi njëkka génn ak palanteer bu 256K-token te mënoon na jëfandikoo lu gëna bari ci benn GPU bu gëna Transformers yuñ mëna méngale, ndax cache KV bi gëna ndaw.
Gis-gis xarala
Mamba xeetu espace-state selective la: ludul bàyyi xel ci bépp token bu weesu, dafay tëye stade recurrent buñ kompresse buy yeesal lineairement ci kaw sequence bi, ak gating bu aju ci dugal biy dogal li ñu wara denc wala fàtte. Jamba dafay boole ay couche yu mat sëkk ci biir couche Mamba yu bari suko defee model bi mëna tëye seetlu bu sori bi gëna bari ci ordinatër bi ak mémoire bi ñuy des ci ligneer, te MoE routing dafay aktive benn subset bu eksper ci token bu nekk.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu xeetu Jamba-Mamba
Hybrid attention ak jëmmal espace state ñu ngi feeñ nekk rëset bu jiitu ci model yu am contexte bu gudd, te Jamba jàppale ci siiwal motif bi. Xaarandi yeneen model yu ubbeeku ak frontiere ngir jël ay stack yu wuute, setal ratio attention-ci-SSM, boole leen ak MoE ak KV-cache tricks. Ginaaw laajte contexte bi dafay gëna yokk ba yegg ci ay milioŋ ciy jeton, njariñu mémoire lineaire bu couches espace-état dafay tax hybrid yi gëna xëcc nit ñi ci aparey yi ak ci deployment yu yomb njëg.
Doxal ci àdduna dëgg
Doxal 256K-token duggal lu melni fichier legal yu gudd wala dencukaay kode yu yaatu ci benn GPU bu mënul méngoo ak Transformer's KV cache
Serwiis chat bu am njariñ bu yaatu, fu nekkinu Mamba di tëye mémoire bi waxtaan yi di gëna màgg
Càmbaru këyitu dokimaa ak defar-yokkum ci xam-xam bu yaatu lool buñ dugal ci muy tekki
Doxal ab LLM bu ubbeeku ci anam wu yaatu (Jamba génnewoon nañu ko ak ay poid yu ubbeeku) ngir gëstu ci architecture hybrid
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Jamba Hybrid Transformer-Mamba Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Modèle spatial d'État et Mamba
Laaj yi ñuy faral di laaj
Jamba Hybrid Transformateur-Mamba Modèles ?
Jamba xeetu làkk bu mag la bu bawoo ci AI21 Labs buy boole ay couche yuy bàyyi xel ci Transformer ak ay couche espace-state Mamba (ak njaxasu eksper yi) ngir am njariñu contexte bu yàgg te baña bàyyi kalite Transformer bi. Dafa am solo ndax dafay wane ni architecture hybrid yi mën nañu raw Transformers yu sell yi ci mémoire ak ci produit yi ci guddaayi sequence yu gudd.
Ban ñaari xeeti core core la Jamba di boole?
Jamba dafay màndargaal màndarga Mamba di dajale ay diisaayu espace-state ak ay diisaayu Transformer yu néew.
Ban beneen pexe la Jamba di jëfandikoo ngir yokk kàttan gi te baña wàññi parametre yi?
Jamba dafay yokk ay couche MoE suko defee benn subset ci eksper yi kese ñooy liggéey ci token bu nekk, di yokk kàttan gi te du yokk ci ordinatër bi.
Lan moo waral Mamba di gëna mëna scale attention ci sequence yu gudd yi?
Mamba dafay denc etaa bu komprime, tolluwaayu fixe buy yeesal ligneer, moytu njëgu bàyyi xel ci quadratic ak cache valeur key buy gëna yokk.
Ban palanteer la xeetu Jamba bu njëkk bi jàppale?
Jamba dafa ubbeeku ak palanteer bu 256K-token, te loolu moo ko mëna def ndax emprent KV-cache bu ndaw bi.
Lu tax Jamba di denc ay couche d'attention moo gën dem Mamba bu sell?
Yenn couche yu mat sëkk dañuy tëye seetlu bu dëggu bi ak kàttan yi ci biir contexte fu model espace-state yu sell yi mëna yeex.