Kugenzura Gradient
Kugenzura Gradient (nanone bita activation checkpointing) ni amayeri yo kubika urwibutso aterera ibikorwa byinshi hagati mugihe cyo gutambuka imbere hanyuma akabisubiza hejuru mugihe cyo gusubira inyuma.
Incamake
It lets you train deeper, larger networks by trading extra compute for much lower memory use.
Kwibira cyane
Imyitozo ngororamubiri isanzwe ibika ibikorwa bya buri cyiciro mugihe cyo gutambuka imbere kuko backpropagation ibakenera kubara gradients. Kuri moderi yimbitse ibyo bikorwa byiganjemo kwibuka. Kugenzura buhoro buhoro ahubwo bizigama ibikorwa gusa kumurongo muto wa 'cheque point' hanyuma ukajugunya ahasigaye. Iyo backprop igeze mukarere ibikorwa byayo byahagaritswe, irongera ikora ibarwa ryambere kugirango icyo gice kibashe kuvugurura ibyo ikeneye, hanyuma ikomeze. Hamwe na bariyeri zashyizwe hafi ya buri kare-imizi-ya-N, ububiko bwibikorwa bigabanuka kuva kuri N gutumiza kare-imizi-ya-N, mugihe compte yazamutse hafi imwe yinyongera yimbere (hafi 20-30% gahoro). Ibi bituma bishoboka guhuza ingano nini nini cyangwa impinduka zimbitse kuri GPU imwe.
Ubushishozi
Tekinike ikoresha igihe-cyo-kwibuka-gucuruza. Kubika ibikorwa byose birihuta ariko ushonje-kwibuka; kubisubiramo bihendutse kubihuta bigezweho ugereranije nigiciro cyo kubura kwibuka. Ibikorwa nka PyTorch (torch.utils.check point) bipfunyika module kugirango ibisohoka byimbere bibike ariko imbere byongeye kubarwa mugihe cyinyuma. Guhitamo igenzurwa ryibintu byingenzi: ndetse no gutandukanya ibice bya sqrt (N) bigabanya ububiko bwuzuye mugihe wongeyeho gusa inzira imwe yimbere yimbere ya compte muri rusange.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza ha Gradient Kugenzura
Kugenzura Gradient ubu birasanzwe mumahugurwa manini-manini kandi biragenda byikora, hamwe namasomero ahitamo ahantu heza kuri wewe. Ihuza bisanzwe na FSDP, ivanze neza, hamwe no gupakurura kugirango usunike urugero rwikigereranyo hejuru. Tegereza 'guhitamo' kugenzura byerekana ibikorwa bihendutse gusa mugihe ubitse ibintu bihenze (nka matrices yo kwitondera), hiyongereyeho uburyo bwo gukusanya ibikoresho mubikoresho nka feri ya PyTorch.compile ihita ihitamo icyo uzigama hamwe no kwishyura kugirango ubone umuvuduko mwiza-wibuke.
Gushyira mu bikorwa Isi
Guhugura transformateur yimbitse ifite ubunini bunini bwicyiciro kuri GPU imwe mu guta no kubara ibikorwa bya layer.
Icyerekezo cyiza cyo kwerekana amashusho kumashusho yikirenga-aho amakarita yo gukora ubundi yarenze ububiko bwa GPU.
Guhobera Guhindura Isura ifasha gradient_checkpointing = Nukuri guhuza miliyari-ibipimo byerekana mugihe cyo gutunganya neza.
Guhuza igenzura hamwe na FSDP kuburyo ibipimo byombi nibikorwa bigumishwa kuba bito, bigafasha amahugurwa yicyitegererezo kinini cyururimi.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Gradient Checkpointing quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Kwiyongera
Ibibazo bikunze kubazwa
What is Gradient Checkpointing?
Kugenzura Gradient (nanone bita activation checkpointing) ni amayeri yo kubika urwibutso aterera ibikorwa byinshi hagati mugihe cyo gutambuka imbere hanyuma akabisubiza hejuru mugihe cyo gusubira inyuma. Iragufasha gutoza byimbitse, imiyoboro minini mugucuruza compte yinyongera yo gukoresha kwibuka cyane.
Ni ubuhe buryo bwo kugenzura buhoro buhoro gucuruza kugirango ubike ububiko?
Kugenzura buhoro buhoro gusubiramo ibikorwa byajugunywe mugihe cyinyuma yinyuma, gukoresha compte yinyongera muguhindura kwibuka cyane.
Ni ukubera iki ibikorwa bisanzwe bibikwa mugihe cyimbere?
Backprop ibara gradients ukoresheje intera intera intera kuva imbere yimbere, bityo igomba kuboneka keretse iyo yongeye kubarwa.
Hafi ya nigute ibikorwa byo kwibuka byibipimo niba igenzura ryashyizwe kuri buri sqrt (N) murwego rwa N-layer?
Kugenzura ibirindiro hafi ya buri kare-imizi-ya-N igabanya ububiko bwibikorwa byabitswe kuva kuri gahunda N kumanuka gutumiza sqrt (N).
Ugereranyije ni bangahe compte yongeyeho neza-igenzura rya gradient igenzurwa mubisanzwe?
Hamwe nogusuzuma neza, hejuru ni hafi yinyongera yimbere, akenshi hafi ya 20-30%.
Muri PyTorch, ni ubuhe butumwa bukoreshwa muburyo bwo kugenzura buhoro buhoro kuri module?
torch.utils.check point ipfunyika module kugirango ibikorwa byayo byimbere bisubizwe mugihe cyinyuma aho kubikwa.