Negative Sampling uye Noise Contrastive Estimation
Negative sampling uye Noise Contrastive Estimation (NCE) matipi anoita kuti mamodheru adzidze pamusoro pemazwi makuru pasina komputa inodhura yakazara softmax.
Pfupiso
Instead of scoring every possible output, they teach the model to tell real (positive) examples from a handful of fake (negative) ones.
Kudzika Kwakadzika
Kana izwi riine mazana ezviuru zvemashoko, yakajairwa softmax inofanirwa kudzikamisa pamusoro peshoko rega rega padanho rega rega rekudzidzisa - kunonoka zvakanyanya. Noise Contrastive Estimation inogadziridza dambudziko sechikamu chebhinari: yakapihwa chinangwa uye mashoma 'ruzha' samples anotorwa kubva kune inozivikanwa kugovera, dzidza kusiyanisa yechokwadi sampuli kubva kune ruzha, iyo inodzoreredza zvinogoneka zvinodikanwa zvingangoitika pasina kujeka kujaira. Negative sampling, yakakurumbira nezwi2vec's skip-gram modhi, ihama yakarerutswa: kune yega yega yechokwadi (izwi, mamiriro) pairi iyo sampuli k zvisizvo uye inodzidzisa modhi kupa yakakwira zvibodzwa kune chaiyo vaviri uye yakaderera zvibodzwa kune iyo fake, uchishandisa sigmoid chinangwa. Ose ari maviri anoshandura dambudziko rinodhura remakirasi akawanda kuita akawanda akachipa mabhinari, zvichiita kuti kudzidziswa kwakakura-kudzika kushande. Sarudzo yekugovera ruzha (kazhinji unigram yakasimudzwa kune iyo 3/4 simba) inokanganisa zvakanyanya kunaka.
Technical Insight
NCE inofungidzira modhi nekuisa dhata maringe neruzha, uye sezvo huwandu hwemasampunzi eruzha hunokura hunogona kuenzanisa mukana wepamusoro-kana ine yakajairwa yakajairwa softmax. Negative sampling inodonhedza mazwi eNCE's normalization zvachose, optimize log σ(positive mamakisi) + Σ log σ(−negative mamakisi). Izvo zvinoita kuti ikurumidze asi isisiri inopindirana density estimator - yakarongedzerwa yekudzidza zvakanaka embeddings pane zvakayerwa zvingangoitika. Sampling negatives kubva kune yakatsetseka unigram kugovera (frequency^0.75) inoyera mazwi akajairika uye asingawanzo.
Strategic Impact
Mutengo uye bhajeti
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
Sarudzo dzakajeka
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
Kudzora kwemhando yepamusoro
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Ramangwana reNegative Sampling uye Noise Contrastive Estimation
Pfungwa yepakati - dzidza nekusiyanisa mapositi kubva kune ega ega - ikozvino inotsigisa yemazuva ano yekuzvitarisira uye inosiyanisa inomiririra kudzidza pane yekuona, mutauro, uye kurudziro. Basa remangwana rinotarisa pane zvakaoma-zvisina kunaka migodhi (kusarudza zvinodzidzisa zvisina kunaka panzvimbo pezvisina kurongeka), kurerekera kune zvisirizvo, uye kuyera zvisina kunaka zvakachipa kuburikidza nehombe ndangariro mabhangi kana mu-batch sampling. Sezvo mamodheru achikura, zvibodzwa zvesampled zvinoshanda zvinoramba zvichikosha chero nzvimbo dzekubuda kana seti dzevamiriri dzakakura, sekudzosa uye vakurudziri vakakura.
Real-World Implementation
word2vec skip-gram ine negative sampling yekudzidza mazwi embeddings kubva kumabhiriyoni ematokeni pasina yakazara softmax.
Mamodheru emitauro kare akashandisa NCE kudzidzisa pamusoro pemashoko emazana ezviuru zvemashoko zvinobudirira.
Kurudziro nekutora masisitimu ekuenzanisira 'negative' zvinhu izvo mushandisi asina kupindirana nazvo kudzidzisa mamodheru ekumisikidza shongwe mbiri.
Girafu uye ruzivo-girafu embeddings (semuenzaniso, kukanganisa musoro kana muswe katatu) uchishandisa sampuli dzisina kunaka kudzidza hukama hwemasangano.
Njodzi & Guardrails
Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Implementation Roadmap
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
Benchmark pasi pechokwadi mutoro uye data mamiriro.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Negative Sampling and Noise Contrastive Estimation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Online uye Zvakaoma Negative Mining
Mibvunzo inowanzo bvunzwa
What is Negative Sampling and Noise Contrastive Estimation?
Negative sampling uye Noise Contrastive Estimation (NCE) matipi anoita kuti mamodheru adzidze pamusoro pemazwi makuru pasina komputa inodhura yakazara softmax. Panzvimbo yekupa zvibodzwa zvese zvinogoneka, vanodzidzisa modhi kutaura chaiyo (yakanaka) mienzaniso kubva kune mashoma emanyepo (asina kunaka).
Idambudziko ripi rinoitwa zvisina kunaka sampling uye NCE zvinonyanya kugadzirisa?
Maitiro ese ari maviri anodzivirira kujairana pamusoro pemazwi makuru nekugadzirisa dzidziso seyakachipa yebhinari rusarura.
Ko Noise Contrastive Estimation inogadziridza sei basa rekudzidza?
NCE inodzidzisa modhi kusiyanisa masampuli chaiwo kubva kumasampuli akatorwa kubva kune inozivikanwa ruzha kugovera.
Ndeupi modhi yakakurumbira sampling isina kunaka yekudzidza kumisikidzwa kwemazwi?
Negative sampling yakaunzwa uye yakakurumbira neskip-gram musiyano wezwi2vec.
Ko sampling yakaipa inosiyana sei neNCE yakazara?
Negative sampling inorerutsa NCE nekubvisa normalization, kutengeserana probabilistic kurongeka kwekumhanya uye zvakanaka embeddings.
Sei zvisina kunaka zvichiwanzo kutorwa kubva pakugovera unigram kusimudzwa kune iyo 3/4 simba?
Iyo 0.75 exponent inotsvedza kugoverwa kwefrequency kuti mazwi akajairika haatongi uye mazwi asingawanzo achiri kuoneka.