Azụmahịa mweghachi nke ịgbalite
Ngụgharị mgbagharị (ntụgharị gradient ma ọ bụ mmalite) na-echekwa ebe nchekwa GPU n'oge ọzụzụ site na ịtụfu mmemme etiti na ngafe mbu na ịtụgharị ha n'oge ngafe azụ.
Nchịkọta
It trades extra compute for the ability to train larger models or longer sequences on the same hardware.
Ime miri emi
Backpropagation chọrọ ọrụ aga-agafe iji gbakọọ gradients, yabụ na ndabara, a na-echekwa ihe nrụpụta oyi akwa ọ bụla - nnukwu ọnụ ahịa ebe nchekwa na-eto na nha ụdị, nha ogbe, na ogologo usoro. Ngụkọta nrụnye na-edobe naanị tenors ole na ole 'checkpoint' (na-abụkarị naanị oke oyi akwa) ma tụfuo ndị ọzọ. N'oge ngafe azụ azụ, ọ na-emegharị ngụkọ ga-aga n'ihu n'etiti ebe nleba anya iji megharịa mmemme ndị a tụfuru na-achọ. Nsonaazụ a kpochapụrụ bụ na ebe a na-enyocha ebe a na-etinye n'ígwé sqrt (N), ebe nchekwa na-adaba na O (sqrt (N)) ka ị na-agbakwunye ihe dị ka otu ngafe na-aga n'ihu (~ 33% karịa). Nhọrọ dị iche iche na-agbakọ naanị ọnụ ala-mana-ncheta-dị arọ (dị ka nlebara anya ma ọ bụ nkwụsị) mgbe ị na-echekwa ndị dị oke ọnụ, na-enweta ọtụtụ n'ime nchekwa nchekwa maka obere nkwụghachi n'elu.
Nghọta nka nka
Azụmaahịa bụ isi bụ ebe nchekwa na FLOPs. Ngụgharị zuru ezu na-agbakwunye otu ngafe na-aga n'ihu kwa nzọụkwụ (~ 30-40% ji nwayọọ nwayọọ) mana ọ nwere ike belata ebe nchekwa ọrụ site n'usoro dị ukwuu. Ntugharị smart bụ nhọrọ nyocha: chọpụta ops ndị buru ibu mana ha dị ọnụ ala (softmax, Layernorm, GELU, akara nlebara anya) wee gbakọọ naanị ndị ahụ, ebe ị na-edobe nsonaazụ GEMM dị oke ọnụ - na-ebelata mkpọkọ efu.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke mmụgharị mkpirisi azụmaahịa
Nhazigharị na-abawanye na akpaaka yana nhọrọ. Frameworks ugbu a na-akọwapụta ebe nchekwa op nke ọ bụla yana ọnụ ahịa FLOP iji họrọ ebe nlele kacha mma, wee jikọta nkwughachi yana mbugharị ọrụ na CPU/NVMe yana yana atụmatụ myirịta. Dị ka ogologo okirikiri na nha ihe nlereanya na-eto eto, na-atụ anya atumatu ndị nchịkọta (na PyTorch, JAX/XLA) na-ahọrọ mkpebi kwa-op na-akpaghị aka, gbakwunyere nchikota nke nkwukọrịta na nkwurịta okwu ka e wee zoo FLOP ndị ọzọ.
Mmejuputa n'ezie n'ụwa
Ịzụ nnukwu transformer nke na-agaghị adabara ma ọ bụghị site na ịlele ngọngọ oyi akwa ọ bụla
Iji PyTorch's torch.utils.checkpoint kechie ihe ngbanwe ma belata ebe nchekwa ọrụ.
Nhọrọ nke nleba anya/softmax na Megatron-LM iji chekwaa ebe nchekwa na obere nwayọ
Na-enyere ogologo usoro ogologo aka na mmefu ego GPU edoziri site na ịgbakọ ọrụ ọrụ kama ịchekwa ha.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Activation Recomputation Tradeoffs quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Quantization SmoothQuant na ịgbalite
Ajụjụ a na-ajụkarị
What is Activation Recomputation Tradeoffs?
Ngụgharị mgbagharị (ntụgharị gradient ma ọ bụ mmalite) na-echekwa ebe nchekwa GPU n'oge ọzụzụ site na ịtụfu mmemme etiti na ngafe mbu na ịtụgharị ha n'oge ngafe azụ. Ọ na-azụta mkpirisi maka ike ịzụ ụdịdị ka ukwuu ma ọ bụ ogologo usoro n'otu ngwaike.
Kedu ihe ngbanwe ngbanwe na-agbanwe iji chekwaa ebe nchekwa?
Nhazigharị na-atụfu mmemme echekwara ma megharịa ha na ngafe azụ azụ, na-emefu mgbakwunye iji belata ojiji ebe nchekwa.
Kedu ihe kpatara eji echekwa mmemme ngafe ma ọlị?
Nfefe azụ azụ na-eji mmegharị mbugharị iji gbakọọ gradients, yabụ na ndabara ha na-edobe na ebe nchekwa ruo mgbe ngafe azụ ga-agba ọsọ.
Ihe dị ka mgbakọ n'ụzọ zuru ezu na-agbakwụnye?
Ngụkọ zuru ezu na-emegharị ngụkọ ga-aga n'ihu n'oge ngafe azụ, na-agbakwunye ihe dị ka otu ngafe ga-aga n'ihu - n'usoro nke 30-40% gbakọọ.
Kedu ihe bụ echiche dị n'azụ nhazigharị nhọrọ (anaghị ezughi oke)?
Nhọrọ ngụkọ ahọpụtara lekwasịrị anya ops ndị na-eji ọtụtụ ebe nchekwa mana obere compute (dị ka softmax ma ọ bụ Layernorm), ebe ị na-echekwa nsonaazụ GEMM dị oke ọnụ iji belata FLOP efu efu.
Kedu usoro nkwado a na-ejikọta ya na nhazigharị iji chekwaa ọbụna ebe nchekwa karịa?
Mbudata arụ ọrụ na-akpali ụfọdụ ịgbalite na nchekwa CPU/NVMe, yana a na-ejikọta ya na ngụkọ na myirịta maka nchekwa nchekwa ọzọ.