Jamba Hybrid Transformer-Mamba Models
Jamba nicyitegererezo kinini cyururimi kuva muri Laboratwari ya AI21 ihuza ibice byitaweho na Transformer hamwe na leta ya Mamba ya leta (hiyongereyeho imvange-yinzobere) kugirango ibone imikorere miremire idatanze ubuziranenge bwa Transformer.
Incamake
It matters because it shows hybrid architectures can beat pure Transformers on memory and throughput at long sequence lengths.
Kwibira cyane
Abahinduye neza bishyura ikiguzi cya quadratic mubitekerezo mugihe imiterere ikura, hamwe nurufunguzo-agaciro cache ballon hamwe nuburebure bwikurikiranya. Icyitegererezo cya leta-umwanya wikigereranyo nka Mamba igipimo cyumurongo kandi ugumane ubunini buhoraho burigihe, ariko mumateka yatinze kwitondera imirimo imwe n'imwe. Jamba ihuza byombi: itondekanya aho usanga ibyiciro byinshi ari Mamba (bihendutse, umurongo, binini kubirebire birebire) kandi umubare muto ni kwitabwaho bisanzwe (bikomeye mubyibutse neza no mubitekerezo). Yongeraho kandi imvange-yinzobere (MoE) ibice kugirango ikure ubushobozi mugihe ikomeza ibipimo byoroheje. Jamba ya mbere yasohoye ifite idirishya rya 256K-token kandi irashobora guhuza imiterere myinshi kuri GPU imwe ugereranije na Transformers igereranywa, bitewe na cache ntoya ya KV.
Ubushishozi
Mamba nicyitegererezo cya leta-umwanya wicyitegererezo: aho kwitabira buri kimenyetso cyashize, ikomeza leta isubirwamo isubirwamo ivugururwa kumurongo ikurikiranye, hamwe ninjiza-ishingiye kumuryango uhitamo icyo ugomba kubika cyangwa kwibagirwa. Jamba ihuza ibice bike byuzuye-byuzuye mubice byinshi bya Mamba kuburyo moderi igumana ibitekerezo byukuri birebire birebire mugihe ibyinshi mubibara hamwe nibuka biguma kumurongo, kandi inzira ya MoE ikora igice cyinzobere kuri buri kimenyetso.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ka Jamba Hybrid Transformer-Mamba Models
Hybrid kwitondera hiyongereyeho ibishushanyo mbonera bya leta bigenda bigaragara nkibisubizo byambere byerekana imiterere ndende-ndende, kandi Jamba yafashije kumenyekanisha icyitegererezo. Tegereza izindi moderi zifunguye kandi zihana imbibi kugirango zemeze ibice bivanze, binonosore ibitekerezo-kuri-SSM, hanyuma ubihuze hamwe na MoE na KV-cache. Nkuko imiterere isaba gukura igera kuri miriyoni yikimenyetso, umurongo-wibuke inyungu ya leta-umwanya wibice bituma ibivange bikurura cyane kubikoresho kandi bikoresha amafaranga.
Gushyira mu bikorwa Isi
Gutunganya 256K-token inyongeramusaruro nka dosiye ndende yemewe cyangwa ububiko bunini bwa kode kuri GPU imwe idashobora guhuza na cache ya Transformer igereranywa
Gukora ibintu byinshi-byinjira-biganira igihe kirekire aho Mamba ihagaze neza igumya kwibuka neza uko ibiganiro bikura
Isesengura ryinyandiko hamwe no kugarura-byongerewe ibisekuru hejuru yubumenyi bunini cyane bwuzuye muburyo bwuzuye
Gukoresha uburemere-burebure-burebure bwa LLM (Jamba yarekuwe afite uburemere bufunguye) kugirango akore ubushakashatsi mubwubatsi bwa Hybrid
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Jamba Hybrid Transformer-Mamba Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Ikirere cya Leta Icyitegererezo na Mamba
Ibibazo bikunze kubazwa
What is Jamba Hybrid Transformer-Mamba Models?
Jamba nicyitegererezo kinini cyururimi kuva muri Laboratwari ya AI21 ihuza ibice byitaweho na Transformer hamwe na leta ya Mamba ya leta (hiyongereyeho imvange-yinzobere) kugirango ibone imikorere miremire idatanze ubuziranenge bwa Transformer. Nibyingenzi kuko byerekana ubwubatsi bwububiko bushobora gutsinda Transformers nziza kububiko no kwinjiza muburebure bukurikiranye.
Ni ubuhe bwoko bubiri bw'ibanze Jamba ihuza?
Jamba isobanura ibiranga ni ugukurikirana Mamba leta-umwanya-hamwe hamwe numubare muto wa Transformer yibitekerezo.
Ni ubuhe buhanga bw'inyongera Jamba akoresha kugirango yongere ubushobozi mugihe ibipimo bikora biri hasi?
Jamba yongeyeho ibice bya MoE kuburyo igice cyinzobere gusa gikora kuri buri kimenyetso, kongerera ubushobozi bwose nta compte ikura.
Ni ukubera iki Mamba igipimo cyiza kuruta kwitondera ibihe birebire?
Mamba ikomeza igabanijwe, ingano-yubunini bwa leta ivugururwa kumurongo, irinda ibiciro bya quadratic yo kwitabwaho hamwe no kwiyongera-urufunguzo-agaciro cache.
Ni ubuhe idirishya ryerekana imiterere ya mbere ya Jamba yashyigikiye?
Jamba yatangije ifite idirishya rya 256K-token, ishobozwa ahanini na KV-cache ntoya.
Kuki Jamba ikomeza kwitondera aho kujya Mamba yera?
Ibice bike byuzuye-byuzuye bigumana isura nyayo nubushobozi-bushingiye aho imiterere-yimiterere yimiterere ishobora gutinda.