I-Stochastic Weight Average
I-Stochastic Weight Averaging (SWA) ithatha isilinganiso esilula sezisindo zemodeli emaphuzwini ambalwa sekwephuzile ekuqeqesheni esikhundleni sokugcina isifinyezo sokugcina.
Uhlolojikelele
This cheap trick often lands the model in a flatter, wider region of the loss landscape, which tends to generalize noticeably better on unseen data.
I-Deep Dive
Yethulwe ngu-Izmailov, uWilson kanye nozakwabo ngo-2018, i-SWA isebenzisa umbono wokuthi i-SGD enezinga lokufunda elingashintshi noma elijikelezayo ayiguquki ibe iphuzu elilodwa - igxumagxuma izungeza unqenqema lwesigodi esibanzi, esiyisicaba. Kunokuba ikhethe enye yalezo zindawo zokuma ezinomsindo, i-SWA isebenzisa izinga lokufunda eliphezulu (ngokuvamile elihlala njalo noma eliwumjikelezo) wezinkathi zokugcina futhi ilinganisela izisindo ezivakashelayo, ngokuvamile yonke inkathi. Izisindo ezimaphakathi zihlala eduze nendawo emaphakathi yendawo eyisicaba. Ngenxa yokuthi izibalo ze-batch-normalization zibalelwa ezisindweni ezithile, i-SWA idinga ukudlula okukodwa okungaphezulu kwedatha ukuze ibale kabusha izindlela ezisebenzayo ze-BN nokuhluka kwemodeli emaphakathi. Izindleko zimahhala, futhi izinzuzo zokunemba ziyahambisana kuzo zonke izihlukanisi zezithombe nangale kwalokho.
I-Technical Insight
I-SWA igcina isilinganiso esisebenzayo w_SWA = (n·w_SWA + w_i)/(n+1) esibuyekezwayo somjikelezo ngamunye, kuyilapho imodeli ye-SGD ebukhoma iqhubeka ihlola ngezinga lokufunda elikhulu uma kuqhathaniswa. Isilinganiso esikhaleni sesisindo silinganisa iqoqo endaweni yokusebenza kodwa kubiza imodeli eyodwa ngencazelo, hhayi eminingi. Indlela eyinhloko ukuthi i-flat minima iqinile ekuphazamisekeni kwesisindo, ngakho izindawo zokuqeqeshwa/ukuhlolwa kokulahlekelwa zihlala ziqondile, kunciphisa igebe elivamile.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa le-Stochastic Weight Average
I-SWA iveze okuhlukile okufana ne-SWA-Gaussian (SWAG) yokungaqiniseki okushibhile kwe-Bayesi, futhi umbono olinganiselayo manje usekela amaqhinga e-Exponential Moving Average asetshenziswa kabanzi kumamodeli asakazwayo, ukufunda ukuzigada, kanye nokuziqeqesha kwamamodeli amakhulu. Lindela ukulinganiswa kwesisindo ukuze kuhlale 'kuyisidlo sasemini' esizenzakalelayo ekuqeqesheni amaresiphi, ngocwaningo olunweba ekuhlanganiseni amamodeli aqeqeshwe ngokuzimela (amamodeli amasobho) kanye nokuthuthukisa ukulinganisa ngokuhambisana nokunemba okungaphekiwe.
Ukuqaliswa Komhlaba Wangempela
Ukuthuthukisa ukunemba kokuhlolwa kwezihlukanisi zezithombe ze-ResNet ne-DenseNet ku-CIFAR ne-ImageNet ngaphandle kwezindleko ezengeziwe.
I-SWAG (SWA-Gaussian) ikhiqiza izilinganiso zokungaqiniseki ezilinganiselwe zokuqagela okuzwelayo zokuphepha kusukela ekugijimeni kokuqeqeshwa okukodwa.
I-EMA-yezisindo iqinisa inethiwekhi yesampula kumajeneretha wesithombe esisabalalisiwe njenge-Stable Diffusion.
Ukwakha 'amasobho emodeli' ngokulinganisa izindawo zokuhlola ezishunwe kahle ukuze kuthuthukiswe ukuqina ngaphandle kokuqeqeshwa kabusha.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Stochastic Weight Averaging quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Isisindo Ukuqalisa
Imibuzo evame ukubuzwa
What is Stochastic Weight Averaging?
I-Stochastic Weight Averaging (SWA) ithatha isilinganiso esilula sezisindo zemodeli emaphuzwini ambalwa sekwephuzile ekuqeqesheni esikhundleni sokugcina isifinyezo sokugcina. Leli qhinga elishibhile livamise ukubeka imodeli endaweni eyisicaba, ebanzi yokwakheka kwezwe yokulahlekelwa, okuvamise ukuvela kangcono kudatha engabonakali.
I-Stochastic Weight Averaging empeleni ilinganisa ini?
I-SWA ilinganisa amapharamitha emodeli (izisindo) eziqoqwe ezindaweni zokuhlola ezimbalwa phakathi nesigaba sokugcina sokuqeqeshwa, hhayi izibikezelo noma ama-gradient.
Kungani i-SWA ijwayele ukuthuthukisa ukujwayelekile?
Ukulinganisa kuyisa isisombululo phakathi nendawo yendawo eyisicaba yendawo yokulahlekelwa, futhi i-flat minima yenza kube ngcono kangcono ngoba ukulahlekelwa kwesitimela nokuhlola kuhlala kuqondile.
Isiphi isinyathelo esengeziwe esisidingayo i-SWA kumanethiwekhi asebenzisa i-batch normalization?
Ngenxa yokuthi izibalo ze-BN zincike ezisindweni ezithile, imodeli emaphakathi idinga ukudlula okukodwa ngaphezulu kwedatha ukuze ibale kabusha izindlela ezisebenzayo nokuhluka.
Ikuphi ukuziphatha kwezinga lokufunda okuvame ukusetshenziswa ngesikhathi sesilinganiso se-SWA?
I-SWA igcina izinga lokufunda elingaguquki eliphezulu noma elijikelezayo ukuze i-SGD iqhubeke nokuhlola indawo eyisicaba ebanzi amaphuzu azo abe eselinganiswa.
Izindleko zokucabanga ze-SWA ziqhathaniswa kanjani nenhlanganisela evamile yamamodeli angu-N?
I-SWA igoqa izindawo zokuhlola eziningi zibe yisethi yesisindo esilinganiselwe, ngakho ukusikisela kubiza okufanayo nemodeli eyodwa kune-N.