Nlereanya na Pipeline Parallelism
Mgbe ihe nlere buru oke ibu ka ọ dabara na otu GPU, ihe nlereanya na myirịta pipeline kewara ihe nlereanya n'onwe ya n'ofe ngwaọrụ.
Nchịkọta
This is what makes training giant language models with hundreds of billions of parameters physically possible.
Ime miri emi
Ihe nlere anya na-ekewa otu ụdị n'ofe GPU dị iche iche, yabụ na ọ nweghị ngwaọrụ kwesịrị ijide ihe niile. Enwere isi ụtọ abụọ. Tensor (intra-layer) myirịta na-ekewa mgbakọ na mwepụ n'ime oyi akwa, dị ka igbutu nnukwu matrix multiplication gafee GPUs nke onye ọ bụla na-agbakọ akụkụ nke mmepụta. Pipeline (inter-layer) parallelism na-ekenye ọkwa dị iche iche na-esochi na GPU dị iche iche, yabụ mgbochi oyibo 1 na-ebi na GPU 0, gbochie 2 na GPU 1, na ihe ndị ọzọ, yana mmemme gafere n'ihu dị ka ahịrị mgbakọ. Ihe ịma aka dị na pipelining naive bụ 'afụ': ebe GPU 0 na-arụ ọrụ na batch mbụ, GPU dị ala na-anọdụ ala. Pipelining na-ekewa ogbe ọ bụla ka ọ bụrụ obere batches ka usoro niile wee na-arụsi ọrụ ike, na-emewanyewanye ojiji n'ụzọ dị egwu.
Nghọta nka nka
Tensor parallelism (dị ka ọ dị na NVIDIA Megatron-LM) na-ekewa kọlụm matrices dị arọ - ma ọ bụ n'ahịrị-amamihe ma na-eji ihe niile belata iji chịkọta nsonaazụ akụkụ, na-edobe nkwukọrịta n'ime oghere NVLink ngwa ngwa. Myirịta pipeline (GPipe, PipeDream) na-ekewa ogbe ahụ ka ọ bụrụ obere batches nke na-agafe n'usoro n'usoro n'usoro, na-ebelata oge 'afụ' na-abaghị uru. A na-ejikọkarị ha abụọ ọnụ, yana myirịta tensor n'ime ọnụ ọnụ yana myirịta pipeline gafee ọnụ ọnụ.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Model na Pipeline Parallelism
Frameworks na-akawanye nsogbu siri ike nke ikpebi ka esi ekewaa ihe nlere n'ofe ngwaọrụ, na-eji profaịlụ na ọchụchọ iji dozie mgbakọ na mwepụ na nzikọrịta ozi. Na-atụ anya njikọ siri ike nke tensor, pipeline, na data parallelism (3D parallelism), nhazi oge micro-batch ka ọ fọrọ nke nta ka ọ ghara ikpochapụ afụ ọkpọkọ, yana ngwaike nwere njikọ ngwa ngwa ka ikewa otu oyi akwa n'ofe ibe ga-adị ọnụ ala ma na-emekarị maka ụdị ndị na-ebuwanye ibu.
Mmejuputa n'ezie n'ụwa
Ụdị ụdị GPT na-enye NVIDIA Megatron-LM ọzụzụ, nke na-ekewa nlebara anya nke oyi akwa ọ bụla yana matrices na-ebugharị n'ofe GPU site na myirịta tensor.
Iji GPipe idowe ọkwa dị iche iche nke nnukwu ọhụụ ma ọ bụ ụdị asụsụ na ngwa ngwa dị iche iche ebe micro-batching na-eme ka ha na-arụsi ọrụ ike.
Igwe pipeline nke DeepSpeed na-ekewa ihe atụ nke ọtụtụ narị ijeri ijeri n'ime usoro n'ofe ọtụtụ ọnụ.
Na-ejikọta tensor parallelism n'ime otu sava 8-GPU nwere myirịta pipeline na-agbasa ọtụtụ sava iji zụọ ihe nlereanya buru ibu maka otu igwe.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Model and Pipeline Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Tensor Parallelism maka Model buru ibu
Ajụjụ a na-ajụkarị
What is Model and Pipeline Parallelism?
Mgbe ihe nlere buru oke ibu ka ọ dabara na otu GPU, ihe nlereanya na myirịta pipeline kewara ihe nlereanya n'onwe ya n'ofe ngwaọrụ. Nke a bụ ihe na-eme ka ọzụzụ nnukwu ụdị asụsụ nwere ọtụtụ narị ijeri paramita n'anụ ahụ kwe omume.
Kedu nsogbu bụ isi ihe nlereanya na myirịta pipeline na-edozi?
Ihe nlere anya na pipeline na-ekewa ihe nlereanya n'onwe ya n'ofe ngwaọrụ, na-enye ọzụzụ nke netwọk dị ukwuu karịa otu GPU ọ bụla nwere ike ijide.
Kedu ka tensor (intra-layer) myirịta si arụ ọrụ?
Tensor parallelism na-ekewa mkpokọta n'ime otu oyi akwa, dịka ọmụmaatụ na-ekewa nnukwu matriks dị arọ ka GPU nke ọ bụla gbakọọ akụkụ nke mmepụta.
Kedu ihe bụ 'afụ' na myirịta pipeline naive?
Na mbido usoro pipeline, usoro mgbada enweghị ntinye ma nọrọ ọdụ na-abaghị uru, na-egbu oge GPU; nke a na-abaghị uru ka a na-akpọ afụ.
Kedu ka myirịta pipeline si ebelata afụ?
Nkewa otu n'ime obere batches na-eme ka ọtụtụ obere obere batches nọrọ n'ọkwa dị iche iche n'otu oge, na-eme ka GPU niile na-arụsi ọrụ ike ma na-ebelata oge efu.
Kedu ihe kpatara eji edebe tensor parallelism n'ime otu ọnụ ọnụ?
Tensor parallelism na-akpakọrịta ugboro ugboro iji jikọta nsonaazụ matriks akụkụ, yabụ etinye ya ebe bandwidth njikọ njikọ kacha elu, n'ime ọnụ n'elu NVLink.