Overview
Jamba imhando yemutauro muhombe kubva kuAI21 Labs inosanganisa Transformer yekutarisisa maseru neMamba state-space layers (pamwe nemusanganiswa-we-nyanzvi) kuti uwane hurefu hwemamiriro ezvinhu pasina kusiya Transformer mhando. Izvo zvine basa nekuti inoratidza ma hybrid architecture anogona kurova yakachena Transformers pandangariro uye kuburikidza nekubuda pahurefu hwakatevedzana hurefu.
Jamba Hybrid Transformer-Mamba Models chikamu chemutauro-AI stack inoshandiswa kuverenga, kugadzira, kuronga, uye kushandura zvinyorwa uye kutaura pamwero.
Deep Dive
Pure Transformers inobhadhara mutengo wequadratic mukutarisisa sezvo mamiriro anokura, uye yavo kiyi-kukosha cache mabharuni ane kutevedzana kureba. Yakachena mamiriro-nzvimbo modhi seMamba chiyero mutsetse uye chengetedza yakagadziriswa-saizi inodzokororwa, asi nhoroondo inoregedza kutarisa kune mamwe mabasa. Jamba inosanganisa ese ari maviri: inoisa zvivharo uko mazhinji akaturikidzana ari Mamba (yakachipa, mutsara, yakanakira kutevedzana kwakareba) uye nhamba diki itarisiro yakajairwa (yakasimba pakurangarira chaiko uye mu-context kufunga). Iyo inowedzerawo musanganiswa-we-nyanzvi (MoE) maseru kuti akure huwandu uku uchichengeta anoshanda ma paramita ane mwero. Yekutanga Jamba yakaburitswa iine 256K-chiratidzo hwindo hwindo uye inogona kukwana zvakanyanya mamiriro paGPU imwechete pane inofananidzwa Transformers, nekuda kweiyo inoshamisa KV cache.
Technical Insight
Mamba isarudzo yenzvimbo-yenzvimbo modhi: pachinzvimbo chekutarisa kune yega yega tokeni, inochengetedza yakadzvanywa inodzokororwa mamiriro akagadziridzwa mutsetse pamusoro pekutevedzana, neinopinza-inotsamira gating inosarudza chekuchengeta kana kukanganwa. Jamba inopindirana mashoma akazara-yekutarisa akaturikidzana pakati peMamba akaturikidzana kuti modhi ichengetedze chaiyo yekutarisa-refu kureba apo yakawanda yekombuta uye ndangariro inoramba iri mutsetse, uye MoE routing inomutsa chete subset yenyanzvi pachiratidzo.
Mastering Jamba Hybrid Transformer-Mamba Models
Jamba imhando yemutauro muhombe kubva kuAI21 Labs inosanganisa Transformer yekutarisisa maseru neMamba state-space layers (pamwe nemusanganiswa-we-nyanzvi) kuti uwane hurefu hwemamiriro ezvinhu pasina kusiya Transformer mhando. Izvo zvine basa nekuti inoratidza ma hybrid architecture anogona kurova yakachena Transformers pandangariro uye kuburikidza nekubuda pahurefu hwakatevedzana hurefu. Jamba Hybrid Transformer-Mamba Models chikamu chemutauro-AI stack inoshandiswa kuverenga, kugadzira, kuronga, uye kushandura zvinyorwa uye kutaura pamwero. Kuti uvake kunzwisisa kwakadzama, tora Jamba Hybrid Transformer-Mamba Models semuenzaniso wekushandisa, kwete chinhu chimwe chete: tsanangura zvinodiwa, kujekesa fungidziro, uye patsanura zvinogona kuitwa nehurongwa hwakavimbika kubva kune izvo zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa Jamba Hybrid Transformer-Mamba Models dhizaini zvinokurudzira, kudzoreredza, uye kuongorora zvishwe seimwe yakabatanidzwa yekutaurirana system. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana. Panguva imwecheteyo, chokwadi cheHallucified chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana kutsvagisa zvinobuda. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
Kugadzira 256K-tokeni yekupinda senge marefu emutemo mafirita kana hombe kodhi repositori paGPU imwechete yaisakwanisa kuenzana neTransformer's KV cache.
Kushandira chat chepamusoro-chepamusoro apo mamiriro eMamba anogara anoramba achiyeuka sezvo hurukuro dzinokura.
Ongororo yezvinyorwa uye kudzoreredza-yakawedzerwa chizvarwa pamusoro pemabhesi makuru eruzivo akaiswa zvakananga muchirevo
Kumhanyisa yakavhurika huremu refu-chinyorwa LLM (Jamba yakaburitswa iine huremu hwakavhurika) yekutsvagisa mune zvakasanganiswa zvivakwa.
Maitiro Ekuita
Jamba Hybrid Transformer-Mamba Models mukuita
Kugadzira 256K-tokeni yekupinda senge marefu emutemo mafirita kana hombe kodhi repositori paGPU imwechete yaisakwanisa kufananidzwa neTransformer's KV cache.
Kugadzirisa 256K-tokeni yekupinda senge mafirita ari pamutemo marefu kana hombe kodhi repositori paGPU imwechete iyo isingakwanise kufananidzwa Transformer's KV cache Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Jamba Hybrid Transformer-Mamba Models mukuita
Kushandira nhaurirano dzepamusoro-soro apo mamiriro eMamba anogara achichengeta ndangariro dzakadzikama sezvo hurukuro dzinokura.
Kushandira yakakwira-yepamusoro-yakareba-chinyorwa chat uko Mamba yakamisikidzwa inochengeta ndangariro dzakadzikama sezvo hurukuro dzichikura Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Jamba Hybrid Transformer-Mamba Models mukuita
Ongororo yezvinyorwa uye kudzoreredza-yakawedzerwa chizvarwa pamusoro pemabhesi makuru eruzivo akaiswa zvakananga muchirevo.
Ongororo yezvinyorwa uye kudzoreredza-yakawedzerwa chizvarwa pamusoro pemabhesi makuru eruzivo akaiswa zvakananga mumamiriro ezvinhu Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura hunhu hwepamberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Jamba Hybrid Transformer-Mamba Models mukuita
Kumhanya yakavhurika-huremu huremu-chinyorwa LLM (Jamba yakaburitswa ine uremu hwakavhurika) yetsvakiridzo mune zvakasanganiswa zvivakwa.
Kumhanyisa yakavhurika huremu-refu-chinyorwa LLM (Jamba yakaburitswa iine huremu hwakavhurika) yekutsvagisa mune zvakasanganiswa zvivakwa Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.