What is Tokenization? Definition | AI Understanding

Maanaam

Xeetu xaaj mbind ci ay jeton ngir dugal ci model.

Term yu ci méngoo

Palanteer ëmbiit

Tolluwaayu dugal bi gëna mag bi benn xeetu làkk mëna def benn yoon.

BPE (kodage ñaari byte)

Algorithme tokenization subword buy boole ñaari araf yi gëna bari ci ay token yuñ mëna jëfandikoowaat.

Koposaŋ bi ci xeetu model biy soppi dugal ci misaal yu nëbbu.

Modèle multimodal

Modèle bu mëna defar wala defar ay xeeti done yu bari lu ci melni bind, nataal wala audio.

OCR (Xamme arafu Optik)

Xarala yuy soppi mbind ci nataal wala skane ci mbind mu masin mëna jàng.

Tokenizer bu xamul làkk, di jàng ay subword ci bind bu ñor te duko njëkka xaaj ci espace bu weex.

Nekk leen ci sunuy tegtal yu amul fayda

SentencePiece Tokenization WordPiece Tokenization Tokenization Subword Tokenization

Xoolal itam

Token Tool Use Temperature Top-k Sampling System Prompt Top-p (Nucleus) Sampling Synthetic Data Transfer Learning

Xoolal Glossaire IA bi yépp Saytu bépp tegtal IA bu amul fayda