Urutonde Kuringaniza no Kwitonda Impeta
Urutonde rukurikiranye rugabanya umurongo umwe muremure winjiza murwego rwa GPU nyinshi kurwego rwa token (igihe), kandi Impeta Yitondewe ireka izo GPU zibara kwitondera neza zinyuze urufunguzo / agaciro bahagaritse impeta.
Incamake
Together they make million-token context windows feasible without any single GPU holding the whole sequence.
Kwibira cyane
Icyitonderwa gisanzwe gikeneye buri kibazo kugirango ubone urufunguzo / agaciro, bityo ibikorwa byo kwibuka bikura hamwe nuburebure bwikurikiranya kandi K / V yuzuye igomba kuboneka. Ikigereranyo kibangikanye kigabanya urukurikirane kuburyo buri GPU ifite igice gihuza ibimenyetso (nibibazo byabo, urufunguzo, indangagaciro). Impeta yitondere noneho itondekanya GPU mumpeta yumvikana: buri gikoresho gikomeza ibibazo byaho bikosorwa mugihe K / V zahagaritswe hop-by-hop kuzenguruka impeta. Mugihe buri gice kigeze, GPU ibara kwitondera igice kandi ikusanya ibisubizo ukoresheje interineti-softmax (kimwe na max / sum trick nka FlashAttention). Nyuma yumuzingi wuzuye, buri kibazo cyitabiriye urufunguzo rwose, nta GPU yigeze ibika K / V. Byibanze, itumanaho rya K / V ryuzuzanya no kubara, bityo ryongeraho igiciro gito cyurukuta-isaha.
Ubushishozi
Impeta Kwitondera gushingira kuri softmax kumurongo: kwitabwaho birashobora kubarwa guhagarika-guhagarika mugihe ukomeza gukora ntarengwa hamwe nogukora ibisanzwe, hanyuma ugakuraho amafaranga igice cyambere mugihe agaciro kanini kagaragaye. Ibi bituma ibisubizo bibarwa muburyo bwo kwitabwaho byuzuye. Impeta inyura gusa kuri K / V (umunzani ufite ubunini, ntabwo ari urukurikirane rwuzuye), kandi kubera ko itumanaho rya buri hop ryuzuzanya na matmul yabanjirije iyi, umurongo mugari - ntabwo ari kwibuka - bihinduka ibintu bigabanya imipaka.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'uruhererekane Kubangikanya no kwitondera impeta
Iringanizwa ryikurikiranya ririmo kuba ihame ryamahugurwa maremare hamwe nu mwanzuro, akenshi bigahuzwa na tensor hamwe nu muyoboro ugereranije na '4D' cyangwa '5D' ugereranije. Ibihinduka nka striped cyangwa zigzag kwitondera kuringaniza akazi katewe no guhisha impamvu. Tegereza impeta-yamenyeshejwe impeta hejuru ya NVLink no kurushaho guhuza hamwe na KV-cache ikuramo, gusunika uburebure bufatika bugera kuri miriyoni mirongo yikimenyetso cyo kugarura, codebase, hamwe ninyandiko ndende.
Gushyira mu bikorwa Isi
Guhugura 1M-token contexte LLM mugukata buri rutonde kuri 8 GPU hamwe nimpeta
Urutonde rwa Megatron-LM kubangikanya kugabanya ububiko bwibikorwa muri LayeriNorm hamwe nabataye ishuri
Gutunganya igitabo cyose cyangwa ububiko bunini bwa kode muri pass imwe yimbere itagabanije
Guhuza Impeta Kwitonda hamwe na tensor parallelism kugirango ihuze ultra-ndende-imiterere-fatizo yerekana kuri GP-node
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sequence Parallelism and Ring Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Tensor Iringaniza Kuri Moderi Nini
Ibibazo bikunze kubazwa
What is Sequence Parallelism and Ring Attention?
Urutonde rukurikiranye rugabanya umurongo umwe muremure winjiza murwego rwa GPU nyinshi kurwego rwa token (igihe), kandi Impeta Yitondewe ireka izo GPU zibara kwitondera neza zinyuze urufunguzo / agaciro bahagaritse impeta. Hamwe na hamwe bakora miriyoni-token imiterere ya windows ishoboka nta GPU nimwe ifata urutonde rwose.
Ni mu buhe buryo uburinganire bukurikirana butandukanya amakuru?
Urukurikirane rwibintu bisobekeranye rukurikirana urutonde rumwe kuri GPU kumurongo wikimenyetso / igihe, buri gikoresho rero gifite igice gihuza ibimenyetso.
Ni ubuhe butumwa Impeta itambuka hagati ya GPU itunganijwe mu mpeta?
Buri GPU ikomeza ibibazo byaho bikosorwa kandi ikanyuza urufunguzo / agaciro guhagarika hop-by-hop kuzenguruka impeta kugirango buri kibazo amaherezo kibone urufunguzo.
Nubuhe buhanga butuma ibitekerezo bibarwa kuburizamo kandi bigahuzwa neza neza?
Kumurongo wa softmax ikurikirana ikigereranyo ntarengwa nigiteranyo, ikuraho ibisubizo byigice nkibikenewe, bigatuma kubara-kubara kubara muburyo bumwe kubitekerezo byuzuye.
Kuberiki Impeta idasaba GPU imwe kugirango ibike K / V yuzuye?
Kuberako K / V ihagarika igera gahoro gahoro kandi buri GPU ikusanya ibitekerezo igice, nta gikoresho na kimwe gikenera urufunguzo / agaciro rwashyizwe mububiko icyarimwe.
Nigute kwitondera Impeta birinda itumanaho kuganza igihe?
Itumanaho rya buri hop ya K / V ryuzuzanya na matrix kugwiza kubiriho, bityo igihe cyo kwimura cyihishe inyuma ya compte.