ZeRO ati Sharded Optimizers
ZeRO (Odo Redundancy Optimizer) imukuro isọdọtun iranti isọnu ti isọdọkan data nipa didin ipo iṣapeye, awọn gradients, ati awọn iwuwo kọja awọn GPUs.
Akopọ
It lets you train enormous models with the simplicity of data parallelism but a fraction of the per-GPU memory.
Jin Dive
Ni afiwe data lasan, gbogbo GPU tọju ẹda kikun ti ipo iṣapeye, awọn gradients, ati awọn paramita, eyiti o jẹ apanirun pupọ, pataki fun Adam, nibiti ipo iṣapeju le jẹ ọpọlọpọ igba iwọn awoṣe funrararẹ. ZeRO, ti a ṣe nipasẹ Microsoft ni DeepSpeed, yọkuro apọju yii kuro nipa pipin awọn tenors wọnyi kọja awọn GPUs ki ẹrọ kọọkan ni bibẹ pẹlẹbẹ nikan. ZeRO wa ni awọn ipele ilọsiwaju mẹta: Ipele 1 ipo iṣapeye shards, Ipele 2 ṣe afikun sharding gradient, ati Ipele 3 shards awọn paramita funrara wọn. Bi o ṣe nilo, GPUs ṣajọ awọn ege ti o padanu nipasẹ ibaraẹnisọrọ, ṣe iṣiro, lẹhinna tu wọn silẹ. Abajade jẹ iranti kekere ti iyalẹnu fun GPU, ṣiṣe awọn bilionu- si ikẹkọ paramita aimọye, lakoko ti o tọju awoṣe siseto irọrun ti afiwe data.
Imọ-imọ-ẹrọ
ZeRO ṣowo ni afikun ibaraẹnisọrọ fun awọn ifowopamọ iranti. Ni Ipele 3, ṣaaju ki o to kọja siwaju Layer kan, gbogbo apejọ n gba awọn aye kikun ti Layer yẹn sori GPU kọọkan; lẹhinna awọn ege ti kii ṣe ohun ini jẹ asonu lati gba iranti pada. Awọn gradients jẹ pipinka-diẹ nitoribẹẹ GPU kọọkan n tọju bibẹ pẹlẹbẹ gradient nikan ni ibaamu awọn aye ti o ni. PyTorch's FSDP (Parded Data Parallel ni kikun) ṣe imuse imọran kanna ni abinibi, awọn modulu murasilẹ si shard ati atunṣatunṣe lori fo.
Ipa Ilana
Iye owo ati isuna
Awọn ipinnu faaji ṣe awakọ iṣẹ ati idiyele iṣẹ fun awọn ọdun.
Awọn ipinnu diẹ sii
Ẹkọ imọ-ẹrọ ṣe iranlọwọ fun awọn ẹgbẹ lati yan akopọ to tọ, kii ṣe ọkan tuntun nikan.
Iṣakoso didara
Awọn yiyan imọ-ẹrọ to dara julọ dinku awọn iṣẹlẹ igbẹkẹle ni iṣelọpọ.
Ojo iwaju ti ZeRO ati Sharded Optimizers
Sharding n di aiyipada fun ikẹkọ iwọn-nla ju aṣayan nla lọ. Reti isọpọ ti o jinlẹ pẹlu piparẹ (titari awọn ege si Sipiyu tabi NVMe nipasẹ ZeRO-Infinity), iṣakojọpọ ti o dara julọ ti gbogbo-kojọpọ ati idinku-tuka pẹlu iṣiro lati tọju idiyele wọn, ati awọn akojọpọ pẹlu tensor ati parallelism pipeline. Bii awọn awoṣe ti n dagba sii, awọn iṣapeye sharded iranti-daradara jẹ aringbungbun si ibamu wọn sori awọn isuna ohun elo ojulowo.
Real-World imuse
Lilo DeepSpeed ZeRO Ipele 2 lati ṣe atunṣe awoṣe ede-ọpọ-bilionu-paramita kan ti yoo bibẹẹkọ ṣaju iranti GPU.
Ikẹkọ pẹlu PyTorch FSDP, eyiti o ṣaja awọn paramita, awọn gradients, ati ipo iṣapeye kọja awọn GPU ati pe o ṣajọ wọn fun Layer lori ibeere.
Nbere ZeRO-Offload lati Titari ipo iṣapeye si iranti Sipiyu, jẹ ki GPU kan ṣe ikẹkọ awoṣe ni ọpọlọpọ igba ti o tobi ju VRAM rẹ lọ.
Awoṣe iwọn aimọye-aimọye kan pẹlu ZeRO-Infinity nipasẹ ṣiṣan paramita shards lati ibi ipamọ NVMe nigbati GPU ati iranti Sipiyu ba jade.
Awọn ewu & Awọn ọna iṣọ
Ṣiṣepe ala-ilẹ kan le tọju awọn ailagbara eto ti o gbooro.
Awọn ohun elo amayederun ati awọn idiyele itọju nigbagbogbo ni aibikita.
Aabo ati awọn ela akiyesi le dagba bi awọn eto ṣe di eka sii.
Ilana Ilana imuse
Ṣetumo lairi, didara, ati awọn ibi-afẹde idiyele ṣaaju imuse.
Aṣepari labẹ ẹru ojulowo ati awọn ipo data.
Abojuto ohun elo fun awọn aṣiṣe, fiseete, ati ipa olumulo.
Mura ipadasẹhin pada ati awọn ipa ọna esi iṣẹlẹ ṣaaju iwọn.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ZeRO and Sharded Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
Lookahead ati kiniun Optimizers
Awọn ibeere ti a beere nigbagbogbo
What is ZeRO and Sharded Optimizers?
ZeRO (Odo Redundancy Optimizer) imukuro isọdọtun iranti isọnu ti isọdọkan data nipa didin ipo iṣapeye, awọn gradients, ati awọn iwuwo kọja awọn GPUs. O jẹ ki o ṣe ikẹkọ awọn awoṣe nla pẹlu ayedero ti isọdọkan data ṣugbọn ida kan ti iranti fun-GPU.
Apọju wo ni ZeRO yọkuro ni akawe si afiwera data itele?
Iparapọ data boṣewa tọju ẹda kikun ti ipo iṣapeye, awọn gradients, ati awọn iwuwo lori gbogbo GPU; ZeRO shards wọnyi nitorina GPU kọọkan di ege kan nikan.
Kini idi ti ipo optimizer nigbagbogbo jẹ hog iranti nla julọ pẹlu Adam?
Adam n ṣetọju awọn iṣiro ṣiṣiṣẹ bii awọn akoko akọkọ ati keji fun paramita, eyiti o ni idapo pẹlu awọn iwọn titunto si fp32 le di iwọn ti ara awoṣe naa.
Kini Shard Ipele 3 ZeRO ti Awọn ipele 1 ati 2 ko ṣe?
Ipele 1 shards optimizer state, Ipele 2 ṣe afikun awọn gradients, ati Ipele 3 lọ siwaju nipasẹ pinpin awọn aye awoṣe kọja awọn GPUs daradara.
Ni Ipele ZeRO 3, bawo ni GPU ṣe gba awọn ayeraye ni kikun ti o nilo fun iwọle siwaju Layer kan?
Ṣaaju ṣiṣe iširo kan Layer, ohun gbogbo-kó assembles awọn oniwe-kikun sile lori kọọkan GPU; ni kete ti o ti ṣe, awọn ege ti kii ṣe ohun-ini ni ominira lati gba iranti pada.
Ẹya PyTorch wo ni o ṣe imuse aṣa Sharding ZeRO?
PyTorch's Fully Sharded Data Parallel (FSDP) paramita shards, gradients, ati ipo iṣapeye, ikojọpọ ati tunto wọn lori fo, ti n ṣe afihan ZeRO.