Skip to content
Dzidza
Nhau
Zvishandiso
Mabasa
Chinangwa
Kutsvaga
⌘K
English
Ipa rubatsiro
Menyu
Tanga pano
Tanga kudzidza
Dzidza
Enda ku Dzidza
Makosi
Nhungamiro
Mudzidzisi weAI
Kurumidza raibhurari
Miedzo
Chitupa
Duramazwi
Nhau
Enda ku Nhau
Nhau dzemazuva ano dzeAI
Musoro wenyaya trackers
Tsvagiridzo & dhatasets
Bhurogu
Zvipimo zvevapepeti
Zvakagadziriswa
Zvishandiso
Enda ku Zvishandiso
Dhairekitori rekushandisa
Zvakanakisisa zvemazita
Enzanisa zvishandiso
Cost Calculator
Prompt Refiner
Tumira chishandiso cheAI
Mabasa
Enda ku Mabasa
Bhurawuza mabasa eAI
Tumira basa
Sponsor AIU
Chinangwa
Enda ku Chinangwa
Chinangwa
Impact & stats
Vanyori
Help Center
Chii chitsva
Rutsigiro
Ipa rubatsiro
Kumba
/
Duramazwi
/
Tokenization
AI Glossary Nguva
Chii
Tokenization
?
Tsanangudzo
Maitirwo ekupatsanura mavara kuita tokens yemhando yekuisa.
Mashoko anoenderana
Context Window
Iyo yakawanda yehuwandu hwekupinza tokeni iyo modhi yemutauro inogona kugadzirisa kamwechete.
BPE (Byte Pair Encoding)
A subword tokenization algorithm iyo inobatanidza anowanzo hunhu pairs kuita reusable tokens.
Encoder
Icho chikamu chemuenzaniso chinoshandura chinopinza kuita chiratidziro chakavanzika.
Multimodal Model
Modhi inogona kugadzirisa kana kugadzira akawanda emhando dzedata senge zvinyorwa, mufananidzo, uye odhiyo.
OCR (Optical Character Recognition)
Tekinoroji inoshandura mavara mumifananidzo kana scanner kuita mavara anoverengwa nemuchina.
SentencePiece
Mutauro-agnostic tokenizer inodzidza subword mauniti zvakananga kubva kune yakaomeswa zvinyorwa pasina pre-kupatsanurwa pachena.
Dzidza zvakawanda mumadhairekitori edu emahara
SentencePiece Tokenization
WordPiece Tokenization
Tokenization
Subword Tokenization
Onawo
Token
Tool Use
Temperature
Top-k Sampling
System Prompt
Top-p (Nucleus) Sampling
Synthetic Data
Transfer Learning
Previous
Token
Next
Tool Use
Bhurawuza yakazara AI Glossary
Ongorora ese emahara AI madhairekitori