What is Token? Definition | AI Understanding

Maanaam

Benn pàccu mbind buñ toppatoo ci misaali làkk, lu ci melni benn baat wala màndarga.

Term yu ci méngoo

Sistem IA yiy defar ëmbiit yu bees yu melni mbind, nataal, audio, wideo wala kod.

Modelu làkk bu mag (LLM)

Benn xeetu làkk buñ tàggat ci corpus mbind yu bari ngir sos ak jàngat mbind.

Modèle multimodal

Modèle bu mëna defar wala defar ay xeeti done yu bari lu ci melni bind, nataal wala audio.

Tokenizer bu xamul làkk, di jàng ay subword ci bind bu ñor te duko njëkka xaaj ci espace bu weex.

Referansu jafe-jafe baat ci math ci daara yu mag yi ñuy jëfandikoo ngir jàngat xalaat jéego ci jéego ci misaali làkk yi.

Xeetu xaaj mbind ci ay jeton ngir dugal ci model.

Nekk leen ci sunuy tegtal yu amul fayda

SentencePiece Tokenization WordPiece Tokenization Tokenizer-Free Byte-Level Models Speculative Streaming and Multi-Token Prediction

Xoolal itam

Temperature System Prompt Tool Use Synthetic Data Top-k Sampling Supervised Learning Top-p (Nucleus) Sampling Sparse Model

Xoolal Glossaire IA bi yépp Saytu bépp tegtal IA bu amul fayda