Model Kantifikaasioŋ
Modèle quantisation dafay wàññi reso neuronal bi ci denc ay nimero ci bit yu néew, suko defee benn model bi di gëna gaaw ci hardware bu gëna ndaw.
Résumé
It is the main reason large models can fit on a single GPU, a laptop, or even a phone.
Plongeur bu xóot
Modèle yiñ tàggat dañuy denc poid bu nekk ci nimero flottant bu 32 wala 16 bit. Kantitasioŋ mooy wecci formaa yi am 8-bit integer (INT8) wala 4-bit valeur (INT4), dagg mémoire bi lu tollu ci 4x ba 8x. Benn model bu am 70 milyaar ci paramet yi soxla lu tollu ci 140GB ci 16-bit mën na wàcci ci 35GB ci 4-bit, mu méngoo ak benn GPU konsomatër. Li gëna am solo mooy njub: soo boole ay valeur yu bari ci 256 wala 16 seau, dafay ñàkk ay detay. Pexe yu bees yu melni GPTQ, AWQ, ak format NF4 yi ñuy jëfandikoo ci QLoRA dañuy tànn facteur scaling yu xarañ yi ba noppi aar poid yi gëna sensitif, moo tax perte kalite bi dafay faral di tuuti. Kantite moo waral jumtukaay yu melni llama.cpp ak Ollama mëna doxal model yu mëna def ci gox bi te amul benn santu done.
Gis-gis xarala
Kantitasioŋ dafay wane valeur dëgg yi ci griy bu ndaw bu amul benn poñ: stored_int = rond (valeur / poñ_nul) + poñ_nul. Tann balans bi bu baax mooy jeu bi yépp. Balaas bu chaine bu nekk wala groupe bu nekk dafay tàqale balance yi ngir daggitu matrix bu diisaay bi, ba noppi di tëye njubte gi ci barab bi war. Post-training quantization dafay soppi model bu jeex, ci noonu la taggat xam-xam quantisation simulate rounding ci diiru taggat suko defee reso bi jàng muñ ko, lu bari ci joxe low-bit accuracy.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu modelu kantite
Xaarandil ni precision bu gëna wàññeeku nekk lu jaadu. Gëstu dafay push 4-bit, 2-bit, ba ci poids binaire yu wóor, boole ci ay pexe yu wuute yuy gëna kawe ay couche yu sensible. Aprey yi ñooy: GPU yi ak puce telefon yi leegi dañuy àndaale ak INT8, INT4, ak FP8 yuñ defaree math. Format yu melni FP8 ak MXFP4 seen mébet mooy boole xeetu flotër yi ak dayo lim yu mat yi. Buñu ko boole ak pexe yu melni QLoRA, kantite dina wéyal def modelu frontier-scale gëna yomb ngir doxal ak defar bu baax ci aparey yu bës bu nekk.
Doxal ci àdduna dëgg
Doxal xeetu Llama 7B wala 13B ci ordinatër portable ak llama.cpp wala Ollama di jëfandikoo fichier GGUF 4-bit.
QLoRA defay ajuste ab model bu rëy ci benn GPU ci tëye poid yu njëkk yi ci 4-bit NF4.
Taxawal xeetu INT8 ci telefon yi am runtimes ci aparey bi suko defee assistant yi liggéey ci offline wala ci seen bopp.
Liggéeyukaay API yu yomb yi nga xamni INT8/FP8 dafay yokk lu tollu ci ñaari yoon produit bi ba noppi wàññi njëgu mémoire bi.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Model Quantization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Modèlu registre
Laaj yi ñuy faral di laaj
What is Model Quantization?
Modèle quantisation dafay wàññi reso neuronal bi ci denc ay nimero ci bit yu néew, suko defee benn model bi di gëna gaaw ci hardware bu gëna ndaw. Moo tax model yu mag yi mëna ànd ak benn GPU, ordinatër portable wala sax telefon.
Lan mooy njëkka soppi model kantite ci reso neuronal bi?
Kantitasioŋ dafay denc benn paramet bi ci bit yu néew (ci misaal INT8 wala INT4 ci barabu flotër 16 wala 32 bit), wàññi mémoire bi ba noppi gaaw math bi.
Lu tollu ci ñaata memory ngay mëna sakkanal soo soppi benn model ci 16-bit dem ci 4-bit?
Joge ci 16 bits ci poids bu nekk dem ba ci 4 bits dafay wàññi dencukaay bi lu tollu ci ñeent, moo tax model yu mag yi mën nañu nekk ci benn GPU.
Lan mooy njariñ li gëna mag ci kwantisation bu am bit bu woyof?
Kartograafi valeur yu bari ci niveau yu diskret yu néew dafay ñàkk ay detay, te loolu mën na wàññi kalite bi fileek scaling bu xarañ bi aar poids yu sensible yi.
Lan moo wuutale tàggat xam-xam kantite ak kantite ginaaw tàggat?
Quantization-xam-xam tàggat yaram tabax njuumte rounding ci biir loop tàggat, suko defee reso bi mëna ànd ak gëna bari njubte ci yaatuwaayu bit yu ndaw.
Ban pexe mooy jëfandikoo 4-bit quantization ngir defar ay model yu yaatu yu yomb ci benn GPU?
QLoRA dafay tëye modelu base bi ci format NF4 4-bit ba noppi di tàggat ay poids adaptateur yu ndaw, may model yu mag yi ñu gëna aju ci hardware bu woyof.