Amamodeli we-Jamba Hybrid Transformer-Mamba
I-Jamba iyimodeli yolimi enkulu evela ku-AI21 Labs ehlanganisa izendlalelo zokunaka ze-Transformer nezendlalelo ze-Mamba state-space (kanye nengxube yochwepheshe) ukuze uthole ukusebenza kahle kokuqukethwe okude ngaphandle kokuyeka ikhwalithi ye-Transformer.
Uhlolojikelele
It matters because it shows hybrid architectures can beat pure Transformers on memory and throughput at long sequence lengths.
I-Deep Dive
Ama-Pure Transformers akhokha inani le-quadratic ekunakeni njengoba umongo ukhula, kanye namabhaluni awo enqolobane yenani elingukhiye anobude bokulandelana. Amamodeli we-state-space ahlanzekile njengesikali se-Mamba ngokulandelana futhi agcina usayizi ongashintshi ophindaphindayo, kodwa ngokomlando abambezeleka kweminye imisebenzi. I-Jamba ihlanganisa kokubili: inqwabelanisa amabhlogo lapho izendlalelo eziningi ziyi-Mamba (ezishibhile, ezinomugqa, zilungele ukulandelana okude) kanye nenombolo encane ukunakwa okujwayelekile (okunamandla ekukhumbuleni okunembayo kanye nokucabanga okungaphakathi kokuqukethwe). Futhi yengeza izendlalelo ze-mix-of-experts (MoE) ukuze kukhule umthamo ngenkathi igcina amapharamitha asebenzayo anesizotha. I-Jamba yokuqala ekhishwe ngefasitela lomongo wethokheni engu-256K futhi ingalingana nomongo omningi ku-GPU eyodwa kune-Transformers eqhathaniswayo, ngenxa yenqolobane yayo ye-KV encane kakhulu.
I-Technical Insight
I-Mamba iyimodeli ye-state-space ekhethiwe: esikhundleni sokubheka yonke ithokheni edlule, igcina isimo esicindezelayo esiphindaphindayo esibuyekezwa ngokulandelana ngokulandelana, ngesango elincike kokufakiwe elinquma ukuthi yini okufanele uyigcine noma uyikhohlwe. I-Jamba ihlanganisa izendlalelo ezimbalwa zokunaka okugcwele phakathi kwezendlalelo eziningi ze-Mamba ukuze imodeli igcine ukubheka okunembile kwebanga elide kuyilapho iningi lekhompuyutha nenkumbulo kuhlala kumugqa, futhi umzila we-MoE wenza kusebenze isethi encane yochwepheshe ngethokheni ngayinye.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa Lamamodeli we-Jamba Hybrid Transformer-Mamba
Ukunakwa okuxubile kanye nemiklamo yesikhala sombuso ivela njengeresiphi ehamba phambili yamamodeli asebenza kahle womongo omude, futhi i-Jamba yasiza ekwenzeni iphethini yaziwe. Lindela amamodeli avuleke kakhulu nangasemngceleni ukuze asebenzise izitaki ezixubile, enze ngcono isilinganiso sokunaka ku-SSM, futhi uwahlanganise namaqhinga e-MoE nawe-KV-cache. Njengoba izimfuno zomongo zikhula zifinyelele ezigidini zamathokheni, inzuzo yenkumbulo yomugqa yezendlalelo zesikhala sikahulumeni yenza ama-hybrid athandeke kakhulu ekusetshenzisweni okukudivayisi nokuzwela izindleko.
Ukuqaliswa Komhlaba Wangempela
Icubungula okokufaka kwamathokheni angu-256K njengokugcwalisa okusemthethweni okude noma amaqoqo ekhodi amakhulu ku-GPU eyodwa engakwazi ukulingana nenqolobane ye-Transformer's KV.
Inikeza ingxoxo yokuqukethwe okuphezulu kokuqukethwe okude lapho isimo se-Mamba esigxilile sigcina inkumbulo iphansi njengoba izingxoxo zikhula
Ukuhlaziywa kwamadokhumenti nokukhiqizwa okuthuthukisiwe kokubuyisa phezu kwezisekelo zolwazi ezinkulu ezigxishwe ngqo kumongo
Isebenzisa i-LLM enesisindo eside esinesisindo esivulekile (i-Jamba ikhishwe ngezisindo ezivulekile) ukuze kwenziwe ucwaningo lwezakhiwo eziyingxubevange.
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Jamba Hybrid Transformer-Mamba Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
State Space Models kanye Mamba
Imibuzo evame ukubuzwa
What is Jamba Hybrid Transformer-Mamba Models?
I-Jamba iyimodeli yolimi enkulu evela ku-AI21 Labs ehlanganisa izendlalelo zokunaka ze-Transformer nezendlalelo ze-Mamba state-space (kanye nengxube yochwepheshe) ukuze uthole ukusebenza kahle kokuqukethwe okude ngaphandle kokuyeka ikhwalithi ye-Transformer. Kubalulekile ngoba kukhombisa ukuthi izakhiwo eziyingxubevange zingashaya ama-Transformer amsulwa kumemori nasekuphumeni ngobude bokulandelana okude.
Yiziphi izinhlobo ezimbili zezingqimba eziwumongo i-Jamba ezizihlukanisa?
Isici esichazayo sika-Jamba sihlanganisa izendlalelo zesikhala se-Mamba kanye nenani elincane lezendlalelo zokunakwa ze-Transformer.
Iyiphi indlela eyengeziwe esetshenziswa i-Jamba ukukhulisa umthamo ngenkathi igcina amapharamitha asebenzayo ephansi?
I-Jamba yengeza izendlalelo ze-MoE ngakho-ke isethi yochwepheshe kuphela esebenzayo ngethokheni ngayinye, ikhulisa umthamo ophelele ngaphandle kokubala ngokukhula ngokulinganayo.
Kungani iMamba ikala kangcono kunokunaka ukulandelana okude?
I-Mamba igcina isimo esicindezelwe, esinosayizi ongashintshi sibuyekezwa ngokulandelana, igwema izindleko zokunakwa kwe-quadratic kanye nenqolobane yenani lokhiye elikhula njalo.
Iliphi iwindi lomongo elasekelwa imodeli yokuqala ye-Jamba?
I-Jamba yethulwe ngefasitela lomongo wethokheni engu-256K, elinikwe amandla kakhulu yi-KV-cache footprint yayo.
Kungani i-Jamba igcina izendlalelo zokunaka kunokuhamba ngeMamba emsulwa?
Izendlalelo ezimbalwa zokunaka okugcwele zigcina ukubukeka okunembile kanye namandla angaphakathi komongo lapho amamodeli we-state-space emsulwa angabambezeleka.