WordPiece Tokenization
WordPiece waa erey-hoosaadka tokenization algorithm kaasoo awood u leh BERT iyo noocyo badan oo Google, u kala qaybiya kelmadaha qaybo dib loo isticmaali karo si qaabku uu u maamulo qoraal kasta oo leh erayo go'an.
Dulmar
It is why a model that has never seen 'unhappiness' can still understand it by reading 'un', '##happy', and '##ness'.
quusid qoto dheer
WordPiece waxay dhistaa ereyada cutubyada kelmad-hoosaadka halkii ay ka dhisi lahayd erayo dhan ama xarfo keliya. Laga bilaabo jilayaasha gaarka ah, waxay si hunguri weyn u midaysaa labada calaamadood ee inta badan kordhiya suurtogalnimada xubnaha tababarka, ku celcelinta ilaa ay ka gaarto cabbirka erayada bartilmaameedka ah (BERT waxay isticmaashaa ilaa 30,000 calaamadood). Marka la eego, waxay calaamadaynaysaa hunguri-xumo bidix-ilaa-midig, oo u dhiganta kelmad-hoosaadka ugu dheer ee erayada, ka dibna ku sii socota inta soo hadhay. Qaybaha sii wadida ee kelmada waxaa lagu calaamadeeyay horgale '##', markaa 'ciyaartu' waxay noqotaa 'ciyaar' + '##ing'. Tani waxay xallisaa dhibka ka baxsan erayada: erayada naadir ah ama aan la arkin waxay si fudud u kala jajabaan jajabyo la yaqaan, ilaa hal xaraf haddii loo baahdo, halka ereyada caadiga ahi ay ahaanayaan hal calaamad oo hufnaan ah.
Aragtida Farsamada
WordPiece way kaga duwan tahay Byte-Pair Encoding marka loo eego cabirkeeda isku darka. BPE waxay isku daraysaa labada lamaane ee ugu badan; WordPiece waxay isku daraysaa lamaanaha kuwaas oo kordhiya suurtogalnimada xogta-tababarka, iyadoo qiyaas ahaan dooranaysa lamaanaha kuwaas oo inta jeer ee isku-dhafan ay ka badan yihiin badeecada qaybaheeda kala duwan. Calaamadeeyaha '##' wuxuu ka soocaa qaybaha hore ee ereyga iyo sii wadida, taasoo u oggolaanaysa calaamadeeyaha inuu dib u dhiso xudduudaha ereyga marka dib loogu celinayo qoraalka.
Saamaynta Istiraatijiyadeed
Xawaaraha iyo miisaanka
Socodka shaqada luqaddu si dhakhso leh ayay u socon kartaa iyada oo aan la hurayn joogteynta.
Helitaanka iyo gaarsiinta
Waxay balaadhisaa gelitaanka luqadaha iyo qaababka isgaarsiinta.
Go'aamo cad
Kooxuhu waxay waqti badan ku qaadan karaan xukunka halka otomaatiggu uu qabanayo ku celcelinta.
Mustaqbalka WordPiece Tokenization
Moodooyinka luqadaha waaweyn ee cusub waxay si isa soo taraysa u door bidaan heerka BPE (qoyska GPT) ama moodooyinka jumlada cutubka, kuwaas oo ka fogaanaya diyaarinta luqadda gaarka ah oo gacanta ku haysa gelinta Unicode kasta. WordPiece ayaa weli ah aasaaska cod-bixiyayaasha BERT-ka-soo-baxa ee weli si ballaaran loo daad gureeyay raadinta iyo kala saarista. Filo isticmaalka joogtada ah ee wax soo saarka NLP, oo ay weheliso cilmi baarista tokenizer-free byte iyo moodooyinka jilayaasha kuwaas oo aakhirka yareyn kara ku tiirsanaanta ereyada hoose ee go'an gabi ahaanba.
Dhaqangelinta Adduunka-dhabta ah
BERT waxay calaameysaa weydiimaha raadinta gudaha Google Raadinta, iyada oo jebinaysa ereyada aan la aqoon kelmado hoose si uu qaabku wali u waafajiyo boggaga khuseeya.
Hugging Face's BertTokenizer waxay isticmaashaa WordPiece si ay qoraalka cayriin ugu beddelato aqoonsiga calaamada ee la quudiyo BERT si loo falanqeeyo dareenka iyo aqoonsiga cidda loo magacaabay.
BERT waxay isticmaashaa erey-bixin la wadaago WordPiece oo ku baahsan 100+ luqadood, taasoo u oggolaanaysa jajabyada in dib loogu isticmaalo qoraallada la xiriira.
DistilBERT iyo kala duwanaanshiyaha BERT ee kiliinikada/biomedical waxay dhaxlaan WordPiece, iyagoo la tacaalaya ereyo caafimaad oo naadir ah sida 'pneumonoconiosis' iyagoo u kala qaybinaya qaybo la yaqaan.
Khatarta & Dariiqyada Ilaalada
Xaqiiqooyinka dhalanteed waxay si deggan u geli karaan warbixinnada, taageerada socodka, ama natiijooyinka cilmi-baarista.
Dareenka degdega ahi wuxuu abuuri karaa natiijooyin aan iswaafaqayn codsiyada la midka ah.
Xogta qoraalka xasaasiga ah ayaa laga yaabaa in la kashifo haddii kontaroolada gelitaanka ay daciif yihiin.
Qorshe Hawleedka Dhaqangelinta
Qeex qaabka wax soo saarka, codka, iyo heerarka tayada ka hor inta aan la baahin.
Jawaabaha salka ku haya ilo lagu kalsoon yahay mar kasta oo saxnidu ay muhiim tahay.
Hayso isbaarada dib u eegista bini aadamka ee wax soo saarka sare.
Lasoco qaababka guuldarada oo dib u leyli dardargelinta ama socodka shaqada si joogto ah.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the WordPiece Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Tokenization Subword
Su'aalaha soo noqnoqda
What is WordPiece Tokenization?
WordPiece waa erey-hoosaadka tokenization algorithm kaasoo awood u leh BERT iyo noocyo badan oo Google, u kala qaybiya kelmadaha qaybo dib loo isticmaali karo si qaabku uu u maamulo qoraal kasta oo leh erayo go'an. Waa sababta qaabka aan waligiis arkin 'farxad-la'aan' uu weli ku fahmi karo akhrinta 'un', '##farxad', iyo '##ness'.
Waa maxay horgalayaasha '##' ku muujinaya wax soo saarka WordPiece?
WordPiece waxay ku calaamadisaa sii wadida kelmad-hoosaadka '##', marka 'ciyaarta' waxay noqotaa 'ciyaar' + '##ing', taasoo u oggolaanaysa qalabeeyaha inuu dib u dhiso xudduudaha ereyga.
Qiyaastii intee bay le'eg tahay erayada WordPiece ee uu isticmaalo BERT-kii asalka ahaa?
BERT waxay isticmaashaa ereybixin WordPiece ah oo qiyaastii ah 30,000 unug kelmad-hoosaad, oo dheellitiraysa daboolida cabbirka-miisaska.
Sidee buu cabbirka isku darka WordPiece uga duwan yahay cod-qaadaha Byte-Pair?
BPE waxay ku biirtaa lamaanaha xiga ee ugu badan, halka WordPiece ay isku darayso lamaanaha kuwaas oo inta badan kordhiya suurtogalnimada jidhka, iyaga oo door bida lammaane kuwaas oo inta jeer ee isku-dhafan ay ka badan yihiin alaabta qaybahooda.
Sidee WordPiece u maamushaa kelmad aan waligeed la arag inta lagu jiro tababarka?
Erayada aan la arkin ayaa loo kala jabiyaa kuwa ugu dheer ee iswaafaqaya ee la yaqaan, iyagoo dib ugu noqda hal jilayaal, kuwaas oo xaliya dhibaatada ereyada ka baxsan.
Waqtiga go'aaminta, sidee WordPiece u doorataa sida erayga loo qaybiyo?
WordPiece waxay u muujisaa hunguri-xumo, oo u dhiganta gelitaanka ereyada ugu dheer ee ka bilaabmaya booska hadda, ka dibna ku celcelinaya inta soo hadhay.