Isampula Elibi kanye Nesilinganiso Sokuqhathanisa Nomsindo
Ukusampula okungalungile kanye ne-Noise Contrastive Estimation (NCE) kungamaqhinga avumela amamodeli afunde kumagama amakhulu ngaphandle kokubala i-softmax egcwele ebizayo.
Uhlolojikelele
Instead of scoring every possible output, they teach the model to tell real (positive) examples from a handful of fake (negative) ones.
I-Deep Dive
Uma silulumagama sinamakhulu ezinkulungwane zamagama, i-softmax evamile kufanele ijwayele phezu kwegama ngalinye esinyathelweni ngasinye sokuqeqesha - ihamba kancane kakhulu. I-Noise Contrastive Estimation ihlela kabusha inkinga njengokwahlukanisa kanambambili: uma kunikezwe ithagethi kanye namasampuli ambalwa 'omsindo' athathwe ekusabalaliseni okwaziwayo, funda ukuhlukanisa isampula yangempela nomsindo, obuyisa ngokusobala amathuba afunekayo ngaphandle kokujwayelekile okusobala. Isampula elibi, elidume ngemodeli ye-word2vec's skip-gram, ingumzala owenziwe lula: kuqiniso ngalinye (igama, umongo) limatanisa amasampula k ama-negative futhi iqeqeshela imodeli ukuthi inikeze amaphuzu aphezulu kubabhangqwana bangempela namaphuzu aphansi kuma-fake, kusetshenziswa inhloso ye-sigmoid. Kokubili kuguqule inkinga ebizayo yezigaba eziningi ibe ngamanani anambambili ashibhile, okwenza ukuqeqeshwa kokushumeka okukhulu kusebenze. Ukukhethwa kokusatshalaliswa komsindo (ngokuvamile i-unigram ephakanyiswe emandleni angu-3/4) kuthinta kakhulu ikhwalithi.
I-Technical Insight
I-NCE ilinganisela imodeli ngokuhlukanisa idatha nomsindo, futhi njengoba inani lamasampuli omsindo likhula cishe licishe libe nokwenzeka kakhulu nge-softmax efanele evamile. Ukusampula okungalungile kwehlisa ngokuphelele imigomo ye-NCE yokujwayela, kuthuthukisa ilogi σ(amaphuzu amahle) + Σ log σ(−amaphuzu angalungile). Lokho kuyenza isheshe kodwa ingabi isilinganiselo sokuminyana esingaguquki - ihlelelwe ukufunda ukushumeka okuhle kunokuthi kube namathuba alinganiselwe. Amasampula amanegethivu asuka ekusabalaliseni kwe-unigram ebushelelezi (imvamisa^0.75) kulinganisa amagama avamile nangavamile.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Lokusampula Okubi kanye Nesilinganiso Esiphikisana Nomsindo
Umbono owumongo - funda ngokuqhathanisa okuhle nokuphikisayo okuyisampula - manje usekela ukufunda kwesimanjemanje kokuziqondisa nokubonisa okungafaniyo kuwo wonke umbono, ulimi, nokuncoma. Umsebenzi wesikhathi esizayo ugxile ezimayini ezimbi (ukukhetha izinto ezingezinhle ezifundisayo esikhundleni salezo ezingahleliwe), ukuhoxisa ama-negative angamanga, kanye nokukala ama-negative ashibhile ngamabhange amakhulu enkumbulo noma amasampula e-in-batch. Njengoba amamodeli ekhula, izinhloso zamasampula ezisebenzayo zihlala zibalulekile nomaphi lapho izikhala zokukhiphayo noma amasethi ekhandidethi zinkulu, njengokubuyisa kanye nezincomo ezinkulu.
Ukuqaliswa Komhlaba Wangempela
I-word2vec skip-gram enesampula engalungile yokufunda amagama ashumekiwe ezigidini zamathokheni ngaphandle kwe-softmax egcwele.
Amamodeli olimi ngokomlando ayesebenzisa i-NCE ukuqeqesha amagama angamakhulu ezinkulungwane zamagama ngendlela efanele.
Amasistimu wokuncoma kanye nokubuyisa amasampula ezinto 'ezimbi' umsebenzisi angazange ahlanganyele nazo ukuze aqeqeshe amamodeli okushumeka emibhoshongo emibili.
Ukushumeka kwegrafu negrafu yolwazi (isb., ukonakalisa ikhanda noma umsila we-triple) kusetshenziswa amasampula angemuhle ukuze kufundwe ubudlelwano bebhizinisi.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Negative Sampling and Noise Contrastive Estimation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Online and Hard Negative Mining
Imibuzo evame ukubuzwa
What is Negative Sampling and Noise Contrastive Estimation?
Ukusampula okungalungile kanye ne-Noise Contrastive Estimation (NCE) kungamaqhinga avumela amamodeli afunde kumagama amakhulu ngaphandle kokubala i-softmax egcwele ebizayo. Esikhundleni sokuthola yonke into engaba khona, bafundisa imodeli ukuthi isho izibonelo zangempela (ezinhle) ezivela kwembalwa yalezo ezingamanga (ezingezinhle).
Iyiphi inkinga isampula engalungile kanye ne-NCE exazulula ngokuyinhloko?
Zombili izindlela zigwema ukujwayela phezu kwesilulumagama esikhulu ngokuhlela kabusha ukuqeqeshwa njengokubandlulula kanambambili okushibhile.
I-Noise Contrastive Estimation iwuhlaka kanjani kabusha umsebenzi wokufunda?
I-NCE iqeqesha imodeli ukuhlukanisa amasampuli wangempela kumasampuli athathwe ekusabalaliseni komsindo okwaziwayo.
Iyiphi imodeli eyenza isampula engalungile yokufunda ishumekwe yamagama?
Ukusampula okungalungile kwethulwa futhi kwaduma ukwehluka kwe-skip-gram kwe-word2vec.
Ingabe isampula elibi lihluke kanjani ku-NCE ephelele?
Ukusampula okungalungile kwenza i-NCE ibe lula ngokususa ukujwayela, ukuhweba ngokunemba okungenzeka ukuze uthole isivinini nokushumeka okuhle.
Kungani ama-negative kuvame ukusampula kusukela ekusabalaliseni kwe-unigram akhushulelwe emandleni angu-3/4?
I-eksponenti engu-0.75 yenza bushelelezi ukusatshalaliswa kwemvamisa ukuze amagama avamile angabuzi futhi amagama ayivelakancane asavela.