Model da Bututu Daidaita
Lokacin da samfurin ya yi girma da yawa don dacewa da GPU ɗaya, ƙira da daidaiton bututun mai suna raba samfurin kanta a cikin na'urori.
Dubawa
This is what makes training giant language models with hundreds of billions of parameters physically possible.
Zurfafa nutsewa
Misalin daidaitaccen tsari yana raba samfuri guda ɗaya a cikin GPUs da yawa don haka babu wata na'ura da ke buƙatar ɗaukar duk ma'aunin nauyi. Akwai manyan abubuwan dandano guda biyu. Tensor (intra-Layer) daidaici yana raba lissafi a cikin Layer, kamar sare babban adadin matrix a cikin GPUs wanda kowane ya ƙididdige ɓangaren fitarwa. Daidaitaccen bututu (inter-Layer) yana ba da nau'i daban-daban a jere zuwa GPUs daban-daban, don haka Layer toshe 1 yana rayuwa akan GPU 0, toshe 2 akan GPU 1, da sauransu, tare da kunnawa gaba kamar layin taro. Kalubalen tare da bututun butulci shine 'kumfa': yayin da GPU 0 ke aiki akan rukunin farko, GPUs na ƙasa suna zama marasa aiki. Bututun bututu yana raba kowane tsari zuwa ƙananan batches don haka dukkan matakai su kasance cikin shagaltuwa, suna haɓaka amfani sosai.
Fahimtar Fasaha
Daidaitawar Tensor (kamar yadda yake a cikin NVIDIA Megatron-LM) yana raba ginshiƙan matrices masu nauyi- ko jere-hikima kuma yana amfani da duk-rage don sake haɗa sakamako na ɓangarori, kiyaye sadarwa a cikin kumburin NVLink mai sauri. Daidaituwar bututun mai (GPipe, PipeDream) yana rarraba tsari zuwa ƙananan batches waɗanda ke gudana ta matakai a cikin jadawali, suna raguwar lokacin 'kumfa' mara amfani. Sau da yawa ana jera su biyun tare, tare da daidaitawar tensor a cikin kulli da daidaiton bututun a fadin nodes.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Model da Daidaituwar Bututu
Tsarin aiki yana ƙara sarrafa matsala mai wuyar yanke shawarar yadda za a raba samfuri a cikin na'urori, ta amfani da bayanin martaba da bincike don daidaita lissafi da sadarwa. Yi tsammanin haɗin kai mai ƙarfi na tensor, bututun, da daidaiton bayanai (daidaitawar 3D), tsara tsarin ƙaramin tsari don kusan kawar da bututun bututun, da kayan masarufi tare da haɗin kai cikin sauri don haka rarraba Layer guda ɗaya a kan kwakwalwan kwamfuta ya zama mai rahusa kuma ƙari na yau da kullun don ƙira mafi girma.
Aiwatar da Gaskiyar Duniya
Horar da nau'ikan salon GPT tare da NVIDIA Megatron-LM, wanda ke raba hankalin kowane Layer na taswira da matrix na ciyar da gaba a cikin GPUs ta hanyar daidaitawa ta tensor.
Yin amfani da GPipe don sanya sassa daban-daban na babban hangen nesa ko ƙirar harshe a kan keɓantattun masu haɓaka yayin da ƙananan batching ke sa su shagaltuwa.
Injin bututun DeepSpeed yana raba samfurin siga-biliyan ɗari zuwa matakai a cikin ƙofofin da yawa.
Haɗa daidaiton tensor a cikin sabar 8-GPU guda ɗaya tare da daidaiton bututun mai da ke faɗin sabar da yawa don horar da ƙira mai girma da yawa ga na'ura ɗaya.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Model and Pipeline Parallelism quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Daidaiton Tensor don Manyan Samfura
Tambayoyin da ake yawan yi
What is Model and Pipeline Parallelism?
Lokacin da samfurin ya yi girma da yawa don dacewa da GPU ɗaya, ƙira da daidaiton bututun mai suna raba samfurin kanta a cikin na'urori. Wannan shi ne abin da ke sa horar da manyan harsunan ƙirar harshe tare da ɗaruruwan biliyoyin ma'auni mai yiwuwa a zahiri.
Wace babbar matsala samfuri da daidaiton bututun mai ke warwarewa?
Samfura da daidaiton bututun mai suna raba samfurin kansa a cikin na'urori, yana ba da damar horar da cibiyoyin sadarwa da yawa fiye da kowane GPU guda ɗaya zai iya ɗauka.
Ta yaya tensor (intra-Layer) daidaici rabo yake aiki?
Tensor parallelism yana raba lissafin a cikin Layer guda ɗaya, misali raba babban matrix nauyi don haka kowane GPU ya ƙididdige ɓangaren fitarwa.
Menene 'kumfa' a daidaici bututun bututu?
A farkon matakin bututun, matakan ƙasa ba su da wani labari tukuna kuma suna zaune a banza, suna ɓata lokacin GPU; ana kiran wannan rata marar aiki da kumfa.
Ta yaya daidaiton bututun zai rage kumfa?
Rarraba tsari zuwa ƙananan batches yana ƙyale ƙananan ƙananan batches da yawa su mamaye matakai daban-daban a lokaci guda, kiyaye duk GPUs cikin aiki da raguwar lokacin zaman banza.
Me yasa aka saba kiyaye tensor parallelism a cikin kulli guda?
Daidaiton Tensor yana sadarwa akai-akai don sake haɗa sakamakon matrix na ɓangaren, don haka ana sanya shi inda haɗin haɗin kai ya fi girma, a cikin kumburi akan NVLink.