Ikojọpọ Gradient
Ikojọpọ Gradient jẹ ki o ṣe adaṣe iwọn ipele nla kan lori iranti GPU ti o lopin nipa pipọ awọn gradients lori ọpọlọpọ awọn ipele kekere kekere ṣaaju mimu dojuiwọn awọn iwọn.
Akopọ
It is the standard workaround for training big models when memory is the bottleneck.
Jin Dive
Ni deede igbesẹ ikẹkọ kan ṣe ilana ipele kan, ṣe iṣiro awọn gradients, ati ṣe imudojuiwọn awọn ayeraye lẹsẹkẹsẹ. Pẹlu ikojọpọ gradient, o ṣiṣe ọpọlọpọ siwaju ati sẹhin kọja lori awọn batches bulọọgi kekere, fifi awọn gradients wọn papọ ni awọn buffers paramita, ati pe nikan ni igbesẹ olupilẹṣẹ (ati odo awọn gradients) lẹhin N micro-batches. Iwọn ipele ti o munadoko di awọn akoko iwọn micro-ipele N, botilẹjẹpe iranti tente oke nikan ni o ni idaduro micro-ipele ti awọn iṣẹ ṣiṣe. Eyi ṣe pataki nitori ọpọlọpọ awọn ilana ikẹkọ gba awọn ipele nla fun awọn iṣiro iduroṣinṣin, ati nitori awọn awoṣe bii awọn oluyipada nla ko le baamu ipele ibi-afẹde ni kikun lori ẹrọ kan. Awọn apeja: ipele-normalization statistiki ti wa ni iṣiro fun bulọọgi-ipele, ki Layer iwuwasi tabi ẹgbẹ iwuwasi bata dara pẹlu ikojọpọ, ati awọn ti o gbọdọ asekale awọn pipadanu ti tọ lati tọju awọn munadoko eko oṣuwọn ọtun.
Imọ-imọ-ẹrọ
Nitori awọn gradients ti ipadanu akopọ jẹ aropo, ikojọpọ awọn gradients lori N micro-batches jẹ mathematiki deede si ipele nla kan, ti o ba jẹ aropin daradara. Awọn imuṣẹ ni igbagbogbo pin ipadanu micro-pipe kọọkan nipasẹ N ṣaaju sẹhin, nitorinaa iwọn didun ti a kojọpọ ṣe dọgbadọgba iwọn lori ipele ti o munadoko ni kikun. O foju optimizer.step () ati zero_grad () titi di Nth micro-batch, iṣowo akoko iṣiro afikun fun iranti tente oke.
Ipa Ilana
Iye owo ati isuna
Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
Awọn ipinnu diẹ sii
Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
Iṣakoso didara
Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ojo iwaju ti ikojọpọ Gradient
Ikojọpọ Gradient yoo duro lefa aiyipada bi awọn iwọn awoṣe ṣe ju iranti ẹrọ ẹyọkan lọ. O npọpọ pọ si pẹlu pipe ti o dapọ, ṣiṣayẹwo imuṣiṣẹ, ZeRO sharding, ati afiwera opo gigun ti epo ni awọn ilana bii DeepSpeed ati FSDP. Reti adaṣe adaṣe ti o muna nibiti awọn igbesẹ ikojọpọ adaṣe-tune awọn ile-ikawe si isuna iranti kan, ati pe o tẹsiwaju pataki fun iṣatunṣe awọn awoṣe nla lori ohun elo iwọntunwọnsi, pẹlu awọn GPU alabara nibiti o ti ṣii ikẹkọ ti bibẹẹkọ ko ṣeeṣe.
Real-World imuse
Titunse awoṣe ede nla kan lori GPU olumulo kan nipa ikojọpọ ju 8 tabi 16 micro-batches lati de ipele ti o munadoko ti awọn ọgọọgọrun.
Ikẹkọ iran ti o ga-giga tabi awọn awoṣe ipin nibiti paapaa ipele ti 2 baamu, ṣugbọn ohunelo nilo ipele ti o munadoko ti 32.
Olukọni Oju Famọra ati Imọlẹ PyTorch ṣe afihan eto gradient_accumulation_steps ti a lo nigbagbogbo ni awọn iṣeto VRAM lopin.
Atunse awọn abajade ipele nla ti iwe kan lori ohun elo kekere nipasẹ ibaamu iwọn ipele ti o munadoko nipasẹ ikojọpọ.
Awọn ewu & Awọn ọna iṣọ
Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Ilana Ilana imuse
Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Gradient Accumulation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
Parẹ ati Bugbamu Gradients
Awọn ibeere ti a beere nigbagbogbo
What is Gradient Accumulation?
Ikojọpọ Gradient jẹ ki o ṣe adaṣe iwọn ipele nla kan lori iranti GPU ti o lopin nipa pipọ awọn gradients lori ọpọlọpọ awọn ipele kekere kekere ṣaaju mimu dojuiwọn awọn iwọn. O jẹ adaṣe boṣewa fun ikẹkọ awọn awoṣe nla nigbati iranti jẹ igo.
Kini ikojọpọ gradient nipataki jẹ ki o ṣe?
Nipa pipọ awọn gradients lori ọpọlọpọ awọn ipele kekere ṣaaju ṣiṣe imudojuiwọn, o ṣe afiwe ipele nla kan laisi idaduro gbogbo rẹ ni iranti ni ẹẹkan.
Lakoko ikojọpọ, nigbawo ni o pe igbesẹ iṣapeye ati odo awọn gradients?
O ṣajọpọ awọn gradients kọja N micro-batches ki o ṣe imudojuiwọn ni ẹẹkan ni opin iyipo naa.
Eyi ti normalization Layer orisii diẹ mọ pẹlu gradient ikojọpọ?
Batch-norm ṣe iṣiro awọn iṣiro fun ipele kekere, nitorinaa Layer tabi iwuwasi ẹgbẹ yago fun ibaamu pẹlu awọn ipele kekere kekere.