Résumé
SmoothQuant xarala la buy may ñu mëna kompresse modeli làkk yu yaatu wàcci ba ci 8-bit integers ngir poids ak activation yépp te kenn du leen tàggataat. Dafa am solo ndax aktivaasioŋ yi ci model yu mag yi dañuy àndaale ak ay outlier yu tar yuy yàq math bu woyof, te SmoothQuant daf leen di yar.
SmoothQuant ak Aktiwisioŋ Kantite ab bloku tabax xarala la buy indi jafe-jafe ci kalite model bi, njëgu infrastructure bi, latency bi, ak wóor ci escalier bi.
Plongeur bu xóot
Soo wàññi benn model ci 16-bit floats dem ba 8-bit integers, poid yi dañuy kompresse ci anam wu yomb waaye activation yi dañuy jafe: yenn chaine yi dañuy yor valeur yu gëna mag 10 ba 100 yoon yeneen yi, te forse leen ñu dugg ci griy integer bu dëgër dafay yàq njub. SmoothQuant, Xiao ak ñeneen ñoo ko dugal. ci 2022, dafa gis ni diisaay yi dañu nooy te yomb nañu xayma ci jamono ji activation yi dañu spiky. Kon ci wàllu math, dafay toxal jafe-jafe bi: dafay xaaj chaine de activation yi ci balance bu chaine bu nekk, ba noppi yokk poid yi ko méngoo ci benn balance bi. Ñaari jëf yooyu dañuy fomm, bàyyi génnug model bi te soppikuwul, waaye leegi ñaari tensor yi dañu toog ci diggante yu xaritoo. Resultaa bi mooy W8A8 (8-bit poids ak activations) inference ak ñàkka am njubte bu jege-zero ak lu tollu ci 2x gaawaay ak sakkanal mémoire.
Gis-gis xarala
Li gëna am solo mooy factëru s buy nooyloo chaine bu nekk ñu xayma ko ni s = max(|X|)^alpha / max(|W|)^(1-alpha). Liggéeyukaay yi dañu leen di eskale ci 1/s ak poid yi ci s, suko defee produit matrix XW bi dafay wéy. Ndax scaling bi dafay absorbé offline ci poids layer bi njëkk wala ab operation fused, du yokk benn coût runtime. Alpha hyperparametre bi (0.5 lu bari) mooy saytu ba ñaata la sargal biy joge ci aktivasioŋ yi dem ci poid yi.
Xam SmoothQuant ak jëfandikoo limu
SmoothQuant xarala la buy may ñu mëna kompresse modeli làkk yu yaatu wàcci ba ci 8-bit integers ngir poids ak activation yépp te kenn du leen tàggataat. Dafa am solo ndax aktivaasioŋ yi ci model yu mag yi dañuy àndaale ak ay outlier yu tar yuy yàq math bu woyof, te SmoothQuant daf leen di yar. SmoothQuant ak Aktiwisioŋ Kantite ab bloku tabax xarala la buy indi jafe-jafe ci kalite model bi, njëgu infrastructure bi, latency bi, ak wóor ci escalier bi. Ngir tabax xam-xam bu xóot, jàppal SmoothQuant ak Activation Quantization ni xeetu liggéey, du benn man-man: leeral njariñ yi nga bëgg, leeral xalaat yi, ak tàqale li sistem bi mëna def ci anam wu wóor ak li ba leegi soxla àtteb kàngam.
Ci jëf, ekip yu am doole yiy jëfandikoo SmoothQuant ak Kantifikaasioŋ ngir gëna baaxal arsitektir, done, ak tànneefi infrastructure ci wàllu wóor ak njëg. Dañuy bind kritër yu leer ngir am ndam, natt leen ci done yu dëggu ak def liggéey, ba noppi ñu baamtu ci anamu ñàkka mëna seetlu, du ci benn yoon benchmark wins. Mooy barab bi xam-xam theorie bi di soppiku nekk kàttan buy yàgg ci produit yi, ci politik yi ak ci liggéey yi.
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw. Ci jamano jooju, Optimisation benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi. Xeetu jëf bi gëna dëgër mooy boole gaawaayu jàngat ak disipline nguur: doxal pilote, jàpp firnde, siiwal dogal yi, ak wéy di yeesal kaaraange gi ci anam wi ñuy doxalee, li jëfandikukat bi di xaar, ak sàrti sàrt yi di jëm kanam.
njeextalu pexe
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw. Ci jëfandikoo yu am kalite bu kawe, loolu dañu koy tekki ci sàrti liggéey yuñ mëna natt, ay peggu boroom, ak ay xew-xewu xoolaat yu bari suko defee ekip yi mëna yokk wóolu seen bopp ci barabu yokk lu jaxasoo.
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal. Ci jëfandikoo yu am kalite bu kawe, loolu dañu koy tekki ci sàrti liggéey yuñ mëna natt, ay peggu boroom, ak ay xew-xewu xoolaat yu bari suko defee ekip yi mëna yokk wóolu seen bopp ci barabu yokk lu jaxasoo.
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi. Ci jëfandikoo yu am kalite bu kawe, loolu dañu koy tekki ci sàrti liggéey yuñ mëna natt, ay peggu boroom, ak ay xew-xewu xoolaat yu bari suko defee ekip yi mëna yokk wóolu seen bopp ci barabu yokk lu jaxasoo.
Doxal ci àdduna dëgg
Liggéeyukaay LLM bu 70B ci W8A8 ci GPU yu néew ci xaaj mémoire bi ak njëgu matrix bi
Aktiwise INT8 ci kaw NVIDIA Hopper/Blackwell core tensor yiy gaawlu math 8-bit
Taxawal ay xeetu waxtaan ci kaw poñ yu mujj yu niir yu am njëg yu bari, fu ñaari yoon li ñuy def di dagg faktiiru token bu nekk
Komprime encodeur transformateur ngir wax wala tekki làkk ci biir aparey, fu kernel 8-bit di gëna gaaw di gëna sedd
Modèlu jëfandikoo
SmoothQuant ak jëfandikoo limu jëfandikoo
Liggéeyukaay LLM-parametre 70B ci W8A8 ci GPU yu néew ci xaaj mémoire ak njëgu matrix.
Liggéeyukaay 70B-parametre LLM ci W8A8 ci GPU yu néew ci xaaj mémoire ak matrix-multiply cost Teams yi dañuy faral di am njariñ yu gëna baax suñu joxee threshold yu baax ci kanam, tëye yoonu escalation nit ngir jafe-jafe yi, ba noppi topp njariñu produit ak njëgu njuumte ci diir bi.
SmoothQuant ak jëfandikoo limu jëfandikoo
Fexe ba INT8 mëna jàppandi ci NVIDIA Hopper/Blackwell tensor yiy gaawlu math 8-bit.
INT8 inference ci NVIDIA Hopper / Blackwell tensor cores yiy gaawlu 8-bit integer math Teams yi dañuy faral di am njariñ yu gëna baax suñu joxee ay pursàntaasu kalite ci kanam, tëye yoonu eskalaasioŋ nit ngir jafe-jafe yi, ba noppi topp njariñu liggéey bi ak njëgu njuumte yi ci diir bi.
SmoothQuant ak jëfandikoo limu jëfandikoo
Taxawal ay xeetu waxtaan ci kaw poñ yu mujj yu cloud yu am njëg yu bari, fu ñaari yoon li ñuy def di dagg faktiiru token bu nekk.
Taxawal xeetu waxtaan ci poñ yu mujj yu niir yu am njëg yu bari, fu ñaari yoon produit yi di dagg ci faktiiru token bu nekk. Ekip yi dañuy faral di am njariñ yu gëna baax suñu joxee ay threshold yu baax ci kanam, tëye yoonu escalation nit ngir jafe-jafe yi, ba noppi topp njariñu produit ak njëgu njuumte ci diir bi.
SmoothQuant ak jëfandikoo limu jëfandikoo
Komprime encodeur transformateur ngir wax ci biir aparey wala tekki làkk fu kernel 8-bit yi gëna gaaw di daw te gëna sedd.
Compresse encodeurs transformateurs ngir wax ci aparey wala tekki fu kernels 8-bit di daw lu gëna gaaw ak sedd. Ekip yi dañuy faral di am njariñ yu gëna baax suñu joxee threshold yu baax ci kanam, tëye yoonu escalation nit ngir jafe-jafe yi, ba noppi topp njariñu produit ak njëgu njuumte ci diir bi.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo. Japp jéego bu nekk ni buntu firnde: sudee mattul kritër yi, noppalu génne gi, tëj bërëb bi, ba noppi yokk jëfandikoo gi.
Benchmark ci biir sargal ak done yu dëggu.
Benchmark ci biir sargal ak done yu dëggu. Japp jéego bu nekk ni buntu firnde: sudee mattul kritër yi, noppalu génne gi, tëj bërëb bi, ba noppi yokk jëfandikoo gi.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi. Japp jéego bu nekk ni buntu firnde: sudee mattul kritër yi, noppalu génne gi, tëj bërëb bi, ba noppi yokk jëfandikoo gi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale. Japp jéego bu nekk ni buntu firnde: sudee mattul kritër yi, noppalu génne gi, tëj bërëb bi, ba noppi yokk jëfandikoo gi.