InoteveraGaidhi rinotevera
Kufurira Mabasa eKudzidzisa Data Attribution
Tekinoroji
Nhungamiro yehunyanzvi
GPTQ uye AWQ Post-Training Quantization
GPTQ neAWQ inzira mbiri dzinotungamira dzekudzikisa mamodheru emitauro yakatodzidziswa kuenda ku4-bit chaiyo saka inomhanya pane yakachipa, diki hardware.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
They are why you can run a capable model on a single consumer GPU instead of a datacenter rack.
Kudzika Kwakadzika
Post-training quantization (PTQ) inodzvanya modhi yakapedzwa isina kuidzidzirazve, kugadzira maremu epamusoro-soro kudzika kusvika ku4 bits kusvika kukota yendangariro. Dambudziko nderekuita izvi pasina kukanganisa chokwadi. GPTQ (kunatsa kweOBQ) inoyera huremu dhizaini nedhizari, uchishandisa yechipiri-odha ruzivo kubva kudiki rekodhi dataset kugadzirisa huremu hwasara uye kubhadhara kukanganisa kwega kwega kutenderedza. AWQ (Activation-aware Weight Quantization) inotora imwe kona: inoona kuti chikamu chidiki chehuremu machanera chakakosha zvisingaenzaniswi, chinoonekwa nekutarisa activation magnitudes, uye inodzivirira idzo nzira dzakasimba nekuyera pane kudziisa zvine hukasha. Ose ari maviri anorega mamodheru akaita seLlama achimhanya mu4-bit, uye zvishandiso zvakaita sevLLM, llama.cpp, uye AutoGPTQ zvaita kuti ive huru kune yemuno uye inodhura-inoshanda inference.
Kuitwa muNyika yeChokwadi
Kumhanyisa 70-bhiriyoni-parameter Llama modhi pane imwechete 24 GB mutengi GPU uchishandisa 4-bit GPTQ huremu.
AWQ-yakaenzana modhi inoshandirwa pakubuda kwepamusoro muvLLM kune inodhura-inoshanda kugadzira APIs.
llama.cpp uchishandisa maremu eGGUF akaverengerwa kumhanyisa mamodheru emitauro munharaunda palaptop yeCPU.
Hugging Face's AutoGPTQ uye AutoAWQ maraibhurari achibvumira vanogadzira kuyera modhi yakatorwa mumitsetse mishoma yekodhi.
Iyo Ramangwana reGPTQ uye AWQ Post-Kudzidzisa Quantization
Quantization iri kusundira pazasi 4 bits kuenda ku3-bit, 2-bit, uye yakasanganiswa-chaiyo zvirongwa, kazhinji zvakasanganiswa ne sparsity. Tarisira kuswedera pedyo nekubatanidza injini dzinoshumira kuitira kuti quantization, KV-cache compression, uye kufungidzira decoding kushanda pamwe chete. Rutsigiro rwe Hardware yeakaderera-bit mafomati seNVFP4 uye MXFP4 iri kukura, uye otomatiki maturusi anowedzera kutora pa-layer bit wides. Chinangwa chakafara chiri pedyo-kurasikirwa 4-bit (uye yakaderera) seyakagadzika, ichiita mamodheru akasimba akachipa kushanda kwese kwese.
Kuzvishandisa
Nei zvichikosha
- Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
- Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
- Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Chenjera
- Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
- Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
- Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Matanho anotevera
- Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
- Benchmark pasi pechokwadi mutoro uye data mamiriro.
- Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
- Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the GPTQ and AWQ Post-Training Quantization quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
What is GPTQ and AWQ Post-Training Quantization?
GPTQ neAWQ inzira mbiri dzinotungamira dzekudzikisa mamodheru emitauro yakatodzidziswa kuenda ku4-bit chaiyo saka inomhanya pane yakachipa, diki hardware. Ndosaka iwe uchigona kumhanyisa modhi inokwanisa pane imwechete mutengi GPU panzvimbo yedatacenter rack.
Chii chinonzi 'post-training quantization' inorevei?
Post-training quantization inoderedza huremu hwemhando yakapedzwa huremu (semuenzaniso, kusvika ku4 bits) pasina kuidzidzira kubva pakatanga.
Nderupi ruzivo rwunoshandiswa neGPTQ kutsiva zvikanganiso zvekutenderedza kana uchiverengera?
GPTQ inoshandisa fungidziro yeHessian kuti inzwisise kuti kuyera huremu humwe kunokanganisa sei kurasikirwa, wozogadzirisa huremu hwasara kuti udzore.
Ndeipi nzwisiso yakakosha inotyaira AWQ (Activation-aware Weight Quantization)?
AWQ inocherekedza huremu chiteshi vachishandisa activation magnitudes uye inovadzivirira kuburikidza nekuyera, sezvo mashoma machanera akakosha zvisingaite.
Ingangove yakawanda sei ndangariro iyo 4-bit quantization inochengetedza maringe ne16-bit uremu?
Kuenda kubva ku16 bits kusvika ku4 bits pahuremu hunocheka huremu hwendangariro kusvika pakota, ingangoita 4x kudzikiswa.
Sei zvese GPTQ neAWQ zvichiwanzo kuyera uremu asi uchichengeta ma activation ari kumusoro chaiko?
Huremu ndiyo huru yekurangarira mutengo, nepo ma activation anonyanya kunzwisiswa nekurasikirwa chaiko, saka huremu-chete quantization ndiyo yakajairika inotapira nzvimbo.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi