Rarraba Gradient
Tarin gradient yana ba ku damar kwaikwayi babban girman tsari akan ƙayyadaddun ƙwaƙwalwar GPU ta hanyar tattara gradients akan ƙananan ƙananan batches da yawa kafin sabunta ma'aunin nauyi.
Dubawa
It is the standard workaround for training big models when memory is the bottleneck.
Zurfafa nutsewa
Normally a training step processes one batch, computes gradients, and immediately updates parameters. Tare da tarin gradient, kuna gudu da yawa gaba da baya akan ƙananan ƙananan batches, kuna ƙara gradients tare a cikin ma'auni, kuma kawai kiran matakin ingantawa (da sifili da gradients) bayan N micro-batches. The effective batch size becomes micro-batch size times N, even though peak memory only ever holds one micro-batch of activations. Wannan yana da mahimmanci saboda yawancin girke-girke na horarwa suna ɗaukar manyan batches don ƙididdige ƙididdiga, kuma saboda samfura kamar manyan taswira ba za su iya dacewa da cikakken tsari na manufa akan na'ura ɗaya ba. Kama: Ana ƙididdige ƙididdige ƙididdiga na al'ada ga kowane ƙaramin batch, don haka ka'ida ko ka'ida ta rukuni sun fi kyau tare da tarawa, kuma dole ne ku daidaita asarar daidai don kiyaye ingantaccen ƙimar koyo daidai.
Fahimtar Fasaha
Saboda gradients na asarar da aka tara suna da ƙari, tara gradients akan N ƙananan batches yana daidai da babban tsari guda ɗaya, idan har kun matsa daidai. Aiwatar da yawanci suna raba kowace asarar micro-batch ta N kafin a koma baya, don haka tarin gradient yayi daidai da ma'ana akan cikakken tsari mai inganci. You skip optimizer.step() and zero_grad() until the Nth micro-batch, trading extra compute time for reduced peak memory.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Tarin Gradient
Gradient accumulation will stay a default lever as model sizes outpace single-device memory. Yana ƙara haɗawa tare da gauraye daidaito, wurin duba kunnawa, ZeRO sharding, da daidaiton bututun mai a cikin tsarin kamar DeepSpeed da FSDP. Yi tsammanin ingantaccen aiki da kai inda ɗakunan karatu ke daidaita matakan tarawa ta atomatik zuwa kasafin kuɗi na ƙwaƙwalwar ajiya, da ci gaba da mahimmanci don daidaita manyan samfura akan kayan aiki masu sauƙi, gami da GPUs na mabukaci inda zai buɗe horon da ba zai yiwu ba.
Aiwatar da Gaskiyar Duniya
Kyakkyawan daidaita babban samfurin harshe akan GPU ɗin mabukaci guda ɗaya ta hanyar tara sama da 8 ko 16 ƙananan batches don isa ga rukunin ɗaruruwa masu inganci.
Horar da babban ƙudurin hangen nesa ko nau'ikan rarrabuwa inda ko da batch na 2 ya dace, amma girke-girke yana buƙatar ingantaccen tsari na 32.
Hugging Face Trainer da PyTorch Walƙiya yana fallasa saitin gradient_accumulation_steps da ake amfani da shi akai-akai a cikin iyakantaccen saitin VRAM.
Sake fitar da babban sakamako na takarda akan ƙaramin kayan aiki ta hanyar daidaita girman tsari mai inganci ta hanyar tarawa.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Gradient Accumulation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Bacewa da Fashe Gradients
Tambayoyin da ake yawan yi
What is Gradient Accumulation?
Tarin gradient yana ba ku damar kwaikwayi babban girman tsari akan ƙayyadaddun ƙwaƙwalwar GPU ta hanyar tattara gradients akan ƙananan ƙananan batches da yawa kafin sabunta ma'aunin nauyi. It is the standard workaround for training big models when memory is the bottleneck.
Menene tarin gradient da farko zai baka damar yi?
Ta hanyar tattara gradients akan ƙananan ƙananan batches da yawa kafin ɗaukakawa, kuna kwaikwayi babban tsari ba tare da riƙe su duka a ƙwaƙwalwar ajiya lokaci ɗaya ba.
Lokacin tarawa, yaushe kuke kiran matakin ingantawa da sifili da gradients?
Kuna tara gradients a fadin N ƙananan batches kuma sabunta sau ɗaya kawai a ƙarshen zagayowar.
Wanne Layer daidaitawa ya haɗa nau'i-nau'i da tsabta tare da tarin gradient?
Batch-al'ada yana ƙididdige ƙididdiga ga kowane ƙaramin batch, don haka ka'ida ko ka'ida ta rukuni yana guje wa rashin daidaituwa tare da ƙananan ƙananan batches.