Imuṣiṣẹ atunkọ Iṣiro Tradeoffs
Iṣiro imuṣiṣẹ (giradient tabi iṣayẹwo imuṣiṣẹ) ṣafipamọ iranti GPU lakoko ikẹkọ nipa sisọnu awọn iṣẹ ṣiṣe agbedemeji ni iwọle siwaju ati ṣiṣatunṣe wọn lakoko iwọle sẹhin.
Akopọ
It trades extra compute for the ability to train larger models or longer sequences on the same hardware.
Jin Dive
Itọpa ẹhin nilo awọn iṣẹ ṣiṣe siwaju-kọja lati ṣe iṣiro awọn iwọn gradients, nitorinaa nipasẹ aiyipada gbogbo awọn abajade ti Layer ti wa ni ipamọ - idiyele iranti nla ti o dagba pẹlu iwọn awoṣe, iwọn ipele, ati gigun ọkọọkan. Iṣiro imuṣiṣẹ ntọju awọn tenors diẹ 'checkpoint' (nigbagbogbo awọn aala Layer) ati ju awọn iyokù lọ. Lakoko iwọle sẹhin, o tun-ṣiṣẹ iṣiro siwaju laarin awọn aaye ayẹwo lati ṣe atunto awọn iṣẹ ṣiṣe ti a danu lori ibeere. Abajade Ayebaye ni pe pẹlu awọn aaye ayẹwo ti a gbe ni gbogbo awọn fẹlẹfẹlẹ sqrt (N), iranti ṣubu si aijọju O (sqrt (N)) lakoko ti o ṣafikun nipa iwe-iwọle siwaju siwaju kan (~ 33% iṣiro diẹ sii). Awọn iyatọ yiyan tun ṣe iṣiro nikan olowo poku-ṣugbọn-iranti-eru ops (bii akiyesi tabi yiyọ kuro) lakoko ti o n ṣafipamọ awọn ti o gbowolori, gbigba pupọ julọ awọn ifowopamọ iranti fun iṣiro pupọ diẹ sii.
Imọ-imọ-ẹrọ
Iṣowo ipilẹ jẹ iranti dipo FLOPs. Iṣiro ni kikun ni aijọju ṣafikun afikun siwaju siwaju ni igbesẹ kan (~ 30-40% losokepupo) ṣugbọn o le ge iranti imuṣiṣẹ nipasẹ aṣẹ titobi. Gbigbe ọlọgbọn jẹ ibi ayẹwo yiyan: ṣe idanimọ awọn ops ti o jẹ iranti-tobi ṣugbọn oniṣiro-olowo poku (softmax, Layernorm, GELU, awọn ikun akiyesi) ati ṣe iṣiro awọn nikan, lakoko ti o tọju awọn abajade ti awọn GEMM ti o gbowolori ti fipamọ - dindinku iṣiro isọnu.
Ipa Ilana
Iye owo ati isuna
Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
Awọn ipinnu diẹ sii
Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
Iṣakoso didara
Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ọjọ iwaju ti Awọn Iṣipopada Iṣiro Iṣiṣẹ ṣiṣẹ
Iṣiro-pada jẹ adaṣe adaṣe pupọ ati yiyan. Awọn ilana ni bayi ṣe profaili iranti op kọọkan ati idiyele FLOP lati yan awọn aaye ayẹwo ti o dara julọ, ati ṣajọpọ iṣiro-iṣiro pẹlu fifisilẹ imuṣiṣẹ si Sipiyu/NVMe ati pẹlu awọn ilana afiwera. Bi awọn ipari ọrọ ọrọ ati awọn iwọn awoṣe ti n dagba sii, nireti awọn eto imulo idari-akopo (ni PyTorch, JAX/XLA) ti o yan awọn ipinnu atunlo-ọkọọkan laifọwọyi, pẹlu isọdọtun ti o pọ ju ti recompute pẹlu ibaraẹnisọrọ ki awọn afikun FLOPs ti wa ni pamọ ni apakan.
Real-World imuse
Ikẹkọ ẹrọ oluyipada nla ti kii yoo baamu bibẹẹkọ nipa ṣiṣayẹwo bulọọki Layer kọọkan
Lilo PyTorch's torch.utils.checkpoint lati fi ipari si awọn bulọọki transformer ati ge iranti imuṣiṣẹ
Iṣiro yiyan ti akiyesi/softmax ni Megatron-LM lati fi iranti pamọ pẹlu idinku kekere
Muu awọn ipari gigun gigun lori isuna GPU ti o wa titi nipa ṣiṣe atunṣiro awọn iṣẹ ṣiṣe dipo fifipamọ wọn
Awọn ewu & Awọn ọna iṣọ
Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Ilana Ilana imuse
Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Activation Recomputation Tradeoffs quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
SmoothQuant ati Iṣatunṣe Quantization
Awọn ibeere ti a beere nigbagbogbo
What is Activation Recomputation Tradeoffs?
Iṣiro imuṣiṣẹ (giradient tabi iṣayẹwo imuṣiṣẹ) ṣafipamọ iranti GPU lakoko ikẹkọ nipa sisọnu awọn iṣẹ ṣiṣe agbedemeji ni iwọle siwaju ati ṣiṣatunṣe wọn lakoko iwọle sẹhin. O ṣe iṣowo iṣiro afikun fun agbara lati ṣe ikẹkọ awọn awoṣe nla tabi awọn ilana to gun lori ohun elo kanna.
Kini atunwi imuṣiṣẹ ṣe iṣowo kuro lati fi iranti pamọ?
Iṣiro-iṣiro sọ awọn iṣẹ ṣiṣe ti o fipamọ silẹ ati tun ṣe wọn ni iwọle sẹhin, ni lilo iṣiro afikun lati dinku lilo iranti.
Kini idi ti awọn imuṣiṣẹ siwaju-kọja ni deede ti o tọju rara?
Ikọja sẹhin nlo awọn imuṣiṣẹ siwaju lati ṣe iṣiro awọn iwọn gradients, nitorinaa nipasẹ aiyipada wọn wa ni iranti titi di igba ti iwọle sẹhin yoo fi ṣiṣẹ.
Ni aijọju melo ni afikun iṣiro ṣe ni kikun ṣiṣiṣẹsẹhin iṣẹ-ṣiṣe ni igbagbogbo ṣafikun?
Iṣiro ni kikun tun-ṣiṣẹ iṣiro siwaju lakoko iwọle sẹhin, fifi aijọju kọja siwaju siwaju - lori aṣẹ ti 30-40% iṣiro diẹ sii.
Kini imọran lẹhin yiyan (kii ṣe kikun) atunlo?
Awọn ibi-iṣiro yiyan yan awọn ops ti o lo ọpọlọpọ iranti ṣugbọn iṣiro kekere (bii softmax tabi Layernorm), lakoko ti o n ṣafipamọ awọn abajade GEMM gbowolori lati dinku awọn FLOP ti o padanu.
Iru ilana ibaramu wo ni igbagbogbo ni idapo pẹlu iṣiro lati fipamọ paapaa iranti diẹ sii?
Imuṣiṣẹpọ pipaṣẹ n gbe diẹ ninu awọn iṣiṣẹ si ibi ipamọ Sipiyu/NVMe, ati pe nigbagbogbo ni idapo pẹlu atunwi ati afiwe fun awọn ifowopamọ iranti siwaju.