Overview
SmoothQuant inzira inoita kuti zvikwanise kudzvanya mamodheru emitauro mikuru kusvika pa8-bit integers kune ese maremu uye ma activation pasina kudzidziswazve. Izvo zvine basa nekuti ma activation mumamodhi mahombe ane akawandisa kunze ayo anowanzo kukanganisa yakaderera-chaiyo math, uye SmoothQuant inovabata.
SmoothQuant uye Activation Quantization inyanzvi yekuvaka inobata mhando yemhando, mutengo wezvivakwa, latency, uye kuvimbika pachiyero.
Deep Dive
Paunoderedza modhi kubva pa16-bit inoyangarara kuenda ku8-bit integers, huremu hunodzvanya nyore asi activations inonetsa: mamwe machanera anotakura zvakakosha 10 kusvika 100 akakura kupfuura mamwe, uye kuamanikidza kupinda coarse integer grid kunoparadza kurongeka. SmoothQuant, yakaunzwa naXiao et al. muna 2022, inoona kuti uremu hwakatsetseka uye huri nyore kuverengera uku activation iri spiky. Saka inotamisa kuoma kwemasvomhu: inopatsanura migero yekugonesa nechiyero che-per-channel uye inowanza huremu hunoenderana nechiyero chimwe chete. Iwo maviri mashandiro akadzima, achisiya iyo modhi yekubuda isina kuchinjika, asi ikozvino ese matensor anogara muhushamwari. Mhedzisiro ndeyeW8A8 (8-bit uremu uye activation) inference ine padyo-zero kurasika kwekurasika uye ingangoita 2x kumhanya uye ndangariro kuchengetedza.
Technical Insight
Iyo yakakosha yehunyengeri ndeye-per-channel yekutsvedza chinhu chinoverengerwa se s = max(|X|)^alpha / max(|W|)^(1-alpha). Activation inoyerwa ne 1/s uye huremu ne s, saka iyo matrix chigadzirwa XW inochengetedzwa. Nekuti kuyera kunopinzwa kunze kweindaneti muhuremu hwekare kana kushanda kwakasanganiswa, inowedzera zero runtime mutengo. Iyo alpha hyperparameter (kazhinji 0.5) inodzora kuti wakawanda sei mutoro wekunze unochinja kubva pakuita kuti uende kune huremu.
Mastering SmoothQuant uye Activation Quantization
SmoothQuant inzira inoita kuti zvikwanise kudzvanya mamodheru emitauro mikuru kusvika pa8-bit integers kune ese maremu uye ma activation pasina kudzidziswazve. Izvo zvine basa nekuti ma activation mumamodhi mahombe ane akawandisa kunze ayo anowanzo kukanganisa yakaderera-chaiyo math, uye SmoothQuant inovabata. SmoothQuant uye Activation Quantization inyanzvi yekuvaka inobata mhando yemhando, mutengo wezvivakwa, latency, uye kuvimbika pachiyero. Kuti uvake kunzwisisa kwakadzama, bata SmoothQuant uye Activation Quantization semuenzaniso wekushandisa, kwete chinhu chimwe chete: tsanangura zvinodiwa, kujekesa fungidziro, uye patsanura izvo zvinogona kuitwa nehurongwa hwakavimbika kubva kune zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa SmoothQuant uye Activation Quantization inokwidziridza zvivakwa, data, uye sarudzo dzezvivakwa zvinopesana nekuvimbika uye mutengo. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore. Panguva imwecheteyo, Kukwirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
Kushandira 70B-parameter LLM paW8A8 pane mashoma maGPU nepakati nepakati ndangariro uye matrix-kuwanza mutengo.
Kugonesa INT8 inference paNVIDIA Hopper/Blackwell tensor cores iyo natively inomhanyisa 8-bit integer math.
Kuendesa machat modhi pane inodhura-inotemerwa makore endpoints uko kupeta kaviri kubuda kunocheka zvakananga bhiri-yechiratidzo.
Kudzvanya maencoder etransformer yekutaura kana kuturikira pamudziyo apo 8-bit kernels inomhanya nekukurumidza uye inotonhorera.
Maitiro Ekuita
SmoothQuant uye Activation Quantization mukuita
Kushandira 70B-parameter LLM paW8A8 pane mashoma maGPU nepakati nepakati ndangariro uye matrix-kuwanza mutengo.
Kushandira 70B-parameta LLM paW8A8 pane mashoma maGPU nepakati nepakati ndangariro uye matrix-kuwanza mutengo Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye kukanganisa mutengo nekufamba kwenguva.
SmoothQuant uye Activation Quantization mukuita
Kugonesa INT8 inference paNVIDIA Hopper/Blackwell tensor cores iyo natively inomhanyisa 8-bit integer math.
Kugonesa INT8 inference paNVIDIA Hopper/Blackwell tensor cores iyo yekuzvarwa inomhanyisa 8-bit integer math Teams kazhinji inowana mhedzisiro iri nani kana ichinge yatsanangura hunhu hwepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
SmoothQuant uye Activation Quantization mukuita
Kutumira machat modhi pamutengo-unomanikidzirwa wegore endpoints uko kupetwa kaviri kuburikidza kunocheka zvakananga bhiri-yechiratidzo.
Kutumira machat modhi pamitengo-inomanikidzirwa yemakore ekupedzisira uko kupetwa kaviri kunocheka zvakananga bhiri-yechiratidziro Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
SmoothQuant uye Activation Quantization mukuita
Kudzvanya maencoder etransformer yekutaura kana kuturikira pamudziyo apo 8-bit kernels dzinomhanya nekuchimbidza uye kuchitonhorera.
Kudzvanya maencoder eshanduro yekutaura pa-mudziyo kana kududzira uko 8-bit kernels inomhanya nekukurumidza uye inotonhorera Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Implementation Roadmap
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Benchmark pasi pechokwadi mutoro uye data mamiriro.
Benchmark pasi pechokwadi mutoro uye data mamiriro. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.