Igikorwa cyo Kwishura Ibicuruzwa
Gusubiramo ibikorwa (gradient cyangwa activation checkpointing) bizigama GPU yibuka mugihe cyamahugurwa ujugunya ibikorwa hagati mugihe cyimbere hanyuma ukabibara mugihe cyinyuma cyinyuma.
Incamake
It trades extra compute for the ability to train larger models or longer sequences on the same hardware.
Kwibira cyane
Backpropagation ikenera ibikorwa-byoherejwe mbere yo kubara gradients, kubwubusanzwe buri cyiciro cyibisubizo byabitswe - ikiguzi kinini cyo kwibuka gikura hamwe nubunini bwikitegererezo, ingano yicyiciro, nuburebure bwikurikiranya. Gusubiramo ibikorwa bigumisha gusa 'cheque point' tensors (akenshi imbibi zurwego) hanyuma ikajugunya ahasigaye. Mugihe cyinyuma cyinyuma, irongera ikora ibarwa yimbere hagati ya bariyeri kugirango igarure ibikorwa byajugunywe kubisabwa. Ibisubizo bya kera ni uko hamwe na bariyeri zashyizwe kuri buri sqrt (N), kwibuka bigabanuka hafi ya O (sqrt (N)) mugihe wongeyeho inzira imwe yinyongera (~ 33% birenze compte). Impinduka zatoranijwe zisubiramo gusa ibintu bihendutse-ariko-kwibuka-biremereye ops (nko kwitabwaho cyangwa guta ishuri) mugihe ufata ibintu bihenze, kubona byinshi mububiko bwibitse kubitari bike cyane.
Ubushishozi
Ubucuruzi bwibanze ni kwibuka na FLOPs. Ibarura ryuzuye ryongeweho hafi yinyongera imbere yintambwe (~ 30-40% gahoro) ariko irashobora kugabanya ibikorwa byo kwibuka ukoresheje gahunda yubunini. Kwimuka kwubwenge ni uguhitamo kugenzura: kumenya ops nini yibuka-nini ariko ikaba ihendutse (softmax, layernorm, GELU, amanota yo kwitondera) hanyuma ugasubiramo ibyo gusa, mugihe ugumya ibisubizo bya GEMMs zihenze - kugabanya compte yapfushije ubusa.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'ibikorwa byo Kwishura ibicuruzwa
Kwishura biragenda byikora kandi birahitamo. Ibikorwa noneho byerekana buri op yibuka na FLOP igiciro kugirango uhitemo igenzura ryiza, kandi uhuze kubara hamwe no gukuramo ibikorwa kuri CPU / NVMe hamwe nuburyo bwo kubangikanya. Nkuko uburebure bwibisobanuro hamwe nubunini bwikitegererezo bikomeza kwiyongera, tegereza politiki ikusanyirizwa hamwe (muri PyTorch, JAX / XLA) itoranya kuri buri op reompute ibyemezo byikora, wongeyeho guhuza cyane na reompute hamwe n’itumanaho bityo FLOPs zihishe igice.
Gushyira mu bikorwa Isi
Guhugura transformateur nini itari guhura nubundi kugenzura buri gice
Koresha itara rya PyTorch.utils.igenzura kugirango uzenguruke uhindure transformateur hanyuma ugabanye ububiko bwibikorwa
Guhitamo guhitamo kwitondera / softmax muri Megatron-LM kugirango ubike ububiko hamwe na buhoro buhoro
Gushoboza uburebure bukurikiranye kuri bije ya GPU ihamye mukongera kubara ibikorwa aho kubibika
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Activation Recomputation Tradeoffs quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Kwiyoroshya no Kuringaniza Ibikorwa
Ibibazo bikunze kubazwa
What is Activation Recomputation Tradeoffs?
Gusubiramo ibikorwa (gradient cyangwa activation checkpointing) bizigama GPU yibuka mugihe cyamahugurwa ujugunya ibikorwa hagati mugihe cyimbere hanyuma ukabibara mugihe cyinyuma cyinyuma. Igurisha compte yinyongera kubushobozi bwo guhugura moderi nini cyangwa urutonde rurerure kubikoresho bimwe.
Ni ubuhe buryo bwo gusubiramo ibicuruzwa bigurisha kure kugirango ubike kwibuka?
Kwisubiraho birukana ibikorwa byabitswe kandi bikabyutsa muburyo bwinyuma, gukoresha compte yinyongera kugirango ugabanye imikoreshereze yibuka.
Ni ukubera iki ibikorwa-byimbere-bikorwa bisanzwe bibikwa muri byose?
Inyuma yinyuma ikoresha ibikorwa byimbere kugirango ibare gradients, kubwubusanzwe bibikwa murwibutso kugeza pass yinyuma ikora.
Hafi yangahe compte yinyongera yongeyeho ibikorwa byuzuye byongera?
Ibarura ryuzuye ryongeye kubara imbere mugihe cyo gusubira inyuma, wongeyeho hafi imwe yinyongera imbere - kurutonde rwa 30-40%.
Ni ikihe gitekerezo kiri inyuma yo gutoranya (kutuzuye)?
Guhitamo kubara guhitamo ops ikoresha ububiko bwinshi ariko kubara bike (nka softmax cyangwa layernorm), mugihe cashe ibisubizo bya GEMM bihenze kugirango ugabanye FLOP yapfushije ubusa.
Ni ubuhe buryo bwuzuzanya bukunze guhuzwa no kubara kugirango ubike kwibuka cyane?
Gukuramo ibintu bimura ibikorwa bimwe mububiko bwa CPU / NVMe, kandi bikunze guhuzwa no kubara no kubangikanya kugirango ubike ububiko bwo kwibuka.