Ikibazo gihuza ibibazo · Biragoye Urwego

Ikibazo Cyihuta na Megatron Amahugurwa Yibibazo

Kugenzura ibyiciro bya Sharing ZeRO, tensor hamwe nu miyoboro ibangikanye, kugenzura ibikorwa, kugenzura no guhuriza hamwe amahugurwa ya 3D abangikanye.

Inzira zijyanye no kuyoboraImyitozo yimbitse na Megatron
Ikibazo 1 Bya 8

Niyihe ntego yibanze ya optimizer ya ZeRO ya DeepSpeed?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Niyihe ntego yibanze ya optimizer ya ZeRO ya DeepSpeed?
    • Kugirango uhindure optimizer leta, gradients, nibipimo muri GPUs kugirango ubike ububiko
    • Guhagarika icyitegererezo cyanyuma cyo kohereza
    • Guhindura icyitegererezo hagati yimikorere
    • Kugirango uhite uranga amakuru yamahugurwa
  2. Tensor parallelism, nkuko yabanjirijwe muri Megatron-LM, igabanya icyitegererezo gute?
    • Mugushira ibice bitandukanye kuri GPU zitandukanye
    • Mugukata matrix kugwiza kugiti cye murwego rwa GPUs
    • Muguha buri GPU icyiciro gitandukanye cyamakuru
    • Kubika ibihe bitandukanye intambwe kuri GPU zitandukanye
  3. Imiyoboro ibangikanye itandukanye na tensor parallelism kuko igabanya icyitegererezo na:
    • Gukata matrices muri buri cyiciro
    • Kwigana icyitegererezo cyose kuri buri GPU
    • Kugenera ibice bitandukanye (ibyiciro) kuri GPU zitandukanye
    • Kugabanya ibiro kugirango ugabanye neza
  4. Niki cyiciro cya ZeRO gitanga ububiko bukomeye bwo kuzigama no kugabanya ibipimo by'icyitegererezo ubwabo?
    • Icyiciro cya 1
    • Icyiciro cya 2
    • Icyiciro 0
    • Icyiciro cya 3
  5. Niki 'activation checkpointing' igurisha kure kugirango ubike kwibuka mugihe cy'amahugurwa?
    • Kubara ibirenze, mukongera kubara ibikorwa mugihe cyinyuma cyinyuma
    • Icyitegererezo
    • Umuyoboro mugari
    • Umubare w'ibyiciro
  6. Kuki guhuza ubwo buhanga bakunze kwita 'parallelism ya 3D'?
    • Ikora gusa ku ikarita ya 3D ishushanya
    • Ihuza amakuru, tensor, hamwe numuyoboro ugereranije hamwe namashoka atatu
    • Ikoresha GPU eshatu ntarengwa
    • Itanga icyitegererezo mubice bitatu
  7. Which ZeRO stage first shards optimizer state across data-parallel workers?
    • Icyiciro 0
    • Icyiciro cya 2
    • Icyiciro cya 1
    • Icyiciro cya 3
  8. What does CPU or NVMe offloading do in large-model training stacks?
    • Copies the full model to every GPU
    • Removes all inter-GPU communication
    • Changes tensor parallelism into reinforcement learning
    • Moves some model or optimizer state off GPU memory to increase the available GPU-memory budget