Usoro Ndekota na mgbanaka Ntị
Usoro myirịta na-ekewa otu usoro ntinye ogologo n'ofe ọtụtụ GPUs n'akụkụ akara (oge), yana Ntị na-eme ka GPU ndị ahụ gbakọọ nlebara anya kpọmkwem site n'ịfefe igodo/ọnụ ahịa gburugburu mgbanaka.
Nchịkọta
Together they make million-token context windows feasible without any single GPU holding the whole sequence.
Ime miri emi
Nlebara anya ọkọlọtọ chọrọ ajụjụ ọ bụla iji hụ igodo / uru ọ bụla, yabụ ebe nchekwa ịgbalite na-eto n'ogologo usoro yana K/V zuru ezu ga-adịrịrị. Usoro myirịta na-emebi usoro nke mere GPU nke ọ bụla nwere ntụtụ akara ngosi (na ajụjụ ha, igodo, ụkpụrụ). Ntị mgbanaka na-ahazi GPU na mgbanaka ezi uche dị na ya: ngwaọrụ ọ bụla na-edobe ajụjụ mpaghara ya ebe a na-agafe ihe mgbochi K/V hop-by-hop gburugburu mgbanaka ahụ. Dị ka ngọngọ ọ bụla na-abịarute, GPU na-agbakọ nlebara anya nke ọma ma na-achịkọta nsonaazụ site na iji online-softmax (otu aghụghọ max/nchikota dị ka FlashAttention). Ka emechara loop zuru oke, ajụjụ ọ bụla abanyela na igodo ọ bụla, na-enweghị GPU na-echekwa K/V dum. N'ụzọ dị mkpa, nkwurịta okwu K/V na-ejikọta na mgbakọ, ya mere ọ na-agbakwụnye obere ego elekere mgbidi.
Nghọta nka nka
Ntị mgbanaka na-adabere na softmax dị n'ịntanetị: enwere ike ịgbakọ nlebara anya ngọngọ-site-block ka ị na-edobe oke na-agba ọsọ yana ihe na-eme ihe na-eme ka ọ na-agba ọsọ, wee na-atụgharị nchikota akụkụ akụkụ mbụ mgbe uru ka ukwuu pụtara. Nke a na-eme ka nsonaazụ mgbakọ na mwepụ yitere na nlebara anya zuru oke. Mgbanaka ahụ na-agafe naanị K / V tenors (nha nha na ngọngọ, ọ bụghị usoro zuru ezu), na n'ihi na nkwurịta okwu hop ọ bụla na-ekpuchi matmul ngọngọ gara aga, bandwit - ọ bụghị ebe nchekwa - na-aghọ ihe na-egbochi ya.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke usoro ndakọrịta na mgbanaka nlebara anya
Myirịta usoro na-aghọ ọkọlọtọ maka ọzụzụ na ntinye okwu ogologo oge, na-ejikọta ya na myirịta tensor na pipeline n'ime okirikiri nhọrọ ukwuu '4D' ma ọ bụ '5D'. Ụdị dị iche iche dị ka nke straipu ma ọ bụ zigzag nlebara anya na-emegharị ọrụ nke ihe nkpuchi kpatara. Na-atụ anya mgbanaka topology maara nke ọma na NVLink yana njikọta siri ike na mwepu KV-cache, na-akwalite ogologo ọnọdụ gburugburu ruo ọtụtụ iri nde token maka iweghachite, codebases na ogologo akwụkwọ.
Mmejuputa n'ezie n'ụwa
Ọzụzụ okirikiri 1M-token LLM site na iji Ntị mgbanaka kesaa usoro nke ọ bụla n'ofe 8 GPU.
Usoro nke Megatron-LM na-ebelata ebe nchekwa ọrụ na LayerNorm na mpaghara nkwụsị
Na-ahazi akwụkwọ dum ma ọ bụ nnukwu ebe nchekwa koodu n'otu ngafe na-aga n'ihu na-enweghị mkpọpu
Ijikọta nleba anya mgbanaka na myirịta tensor ka dabara na ntụnye okwu ogologo ogologo na ọnụ ọtụtụ GPU
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Sequence Parallelism and Ring Attention quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Tensor Parallelism maka Model buru ibu
Ajụjụ a na-ajụkarị
What is Sequence Parallelism and Ring Attention?
Usoro myirịta na-ekewa otu usoro ntinye ogologo n'ofe ọtụtụ GPUs n'akụkụ akara (oge), yana Ntị na-eme ka GPU ndị ahụ gbakọọ nlebara anya kpọmkwem site n'ịfefe igodo/ọnụ ahịa gburugburu mgbanaka. Ha jikọtara ọnụ na-eme ka nde-token okirikiri windo ga-ekwe omume na-enweghị otu GPU na-ejide usoro niile.
N'akụkụ nke akụkụ ka usoro myirịta na-ekewa data ahụ?
Usoro myirịta usoro na-akụda otu usoro n'ofe GPUs n'akụkụ axis akara/oge, yabụ ngwaọrụ ọ bụla nwere ntụtụ akara ngosi.
Kedu ihe nleba anya mgbanaka na-agafe n'etiti GPU ndị ahaziri na mgbanaka?
GPU nke ọ bụla na-edobe ajụjụ mpaghara ya ma na-agafe igodo / uru na-egbochi hop-by-hop gburugburu mgbanaka ka ajụjụ ọ bụla mechaa hụ igodo ọ bụla.
Kedu usoro na-ahapụ nlebara anya gbakọọ ngọngọ-site-ngbochi ma ka dabara nlebara anya zuru oke?
Ngwa softmax dị n'ịntanetị na-akwado oke na nchikota na-agba ọsọ, na-eweghachi nsonaazụ akụkụ dị ka ọ dị mkpa, na-eme ka kọmpụta nke maara ihe na-eme ka ọnụọgụgụ dị ka otu na nlebara anya zuru oke.
Kedu ihe kpatara nlebara anya mgbanaka anaghị achọ GPU ọ bụla iji chekwaa K/V zuru ezu?
N'ihi na ihe mgbochi K/V na-abata n'ụzọ na-arịwanye elu na GPU ọ bụla na-anakọta nlebara anya nke ọma, ọ dịghị ngwaọrụ ọ bụla chọrọ igodo / uru niile edobere na ebe nchekwa ozugbo.
Kedu ka nlebara anya mgbanaka si egbochi nzikọrịta ozi ka ọ ghara ịchịkwa oge ojiri gaa?
Nkwukọrịta K/V nke ọ bụla na-ejikọta ya na ọnụọgụ matriks maka ngọngọ dị ugbu a, yabụ oge mbufe na-ezoro ezo n'azụ mgbakọ.