Kedis ku xidhan hagaha · Adag Heerka
DeepSpeed iyo Megatron Tababbarrada Tababarka Kediska
Waxay hubisaa marxaladaha jeexjeexa ZeRO, isbarbardhigga tensor-ka iyo dhuumaha, isbaarada hawlgelinta, rarista iyo isku-dhafka 3D tababbarka barbar socda.
Su'aal 1 ee 8
Waa maxay ujeedada koowaad ee DeepSpeed's ZeRO optimizer?
Horudhac su'aal kasta8 su'aalo
Jawaabaha iyo sharraxaadaha waxay soo baxayaan markaad qaadanayso kediska.
- Waa maxay ujeedada koowaad ee DeepSpeed's ZeRO optimizer?
- Si loo jeexjeexo hagaajinta gobolada, gradients, iyo cabbirada guud ahaan GPU-yada si loo badbaadiyo xusuusta
- Si loo cadaadiyo qaabka ugu dambeeya ee dirida
- Si loo turjumo moodooyinka u dhexeeya qaab-dhismeedka
- Si toos ah loogu calaamadiyo xogta tababarka
- Isbarbardhigga Tensor-ka, sidii horudhac ugu ahaa Megatron-LM, u kala qaybiyaa qaabka sidee?
- Iyadoo la saarayo lakabyo kala duwan GPU-yada kala duwan
- Iyada oo la jarjarayo isku dhufashada matrixka shakhsi ahaaneed ee lakabka guud ahaan GPU-yada
- Adigoo siinaya GPU kasta xog kala duwan
- Adigoo ku kaydinaya tillaabooyin waqtiyo kala duwan GPU-yada kala duwan
- Isbarbar yaaca tuubooyinka way ka duwan yihiin isbarbardhigga tensor sababtoo ah waxay u kala qaybisaa moodalka:
- Goynta matrices gudaha lakab kasta
- Ku koobida moodalka oo dhan GPU kasta
- U qoondaynta lakabyo kala duwan (marxalado) GPU-yo kala duwan
- Xisaabinta miisaanka si loo hooseeyo saxnaanta
- Marxaladee ZeRO ayaa bixisa kaydinta xusuusta ugu weyn iyaga oo sidoo kale kala qaybinaya cabbirada tusaalaha laftooda?
- Marxaladda 1
- Marxaladda 2
- Marxaladda 0
- Marxaladda 3
- Maxay 'baaritaan hawlgelinta' ka beddeshaa si loo badbaadiyo xusuusta inta lagu jiro tababarka?
- Xisaabinta dheeraadka ah, adoo dib u xisaabinaya hawlqabadyada inta lagu jiro kaarka dambe
- Saxnaanta moodeelka
- bandwidth network
- Tirada lakabyada
- Waa maxay sababta isku-darka farsamooyinkan inta badan loogu yeero 'isbarbardhigga 3D'?
- Waxay kaliya ku shaqeysaa kaararka garaafyada 3D
- Waxay isku daraysaa xogta, tensorka, iyo isbarbardhigga dhuumaha oo ay weheliyaan saddex faas
- Waxay isticmaashaa saddex GPU-yada ugu badan
- Waxay ku soo bandhigtaa moodooyinka saddex cabbir
- Which ZeRO stage first shards optimizer state across data-parallel workers?
- Marxaladda 0
- Marxaladda 2
- Marxaladda 1
- Marxaladda 3
- What does CPU or NVMe offloading do in large-model training stacks?
- Copies the full model to every GPU
- Removes all inter-GPU communication
- Changes tensor parallelism into reinforcement learning
- Moves some model or optimizer state off GPU memory to increase the available GPU-memory budget