What is Token? Definition | AI Understanding

Tsanangudzo

Chikamu chemavara chinogadziriswa nemhando dzemitauro, senge chidimbu chezwi kana chiratidzo.

Mashoko anoenderana

AI masisitimu anoburitsa zvinyorwa zvitsva senge zvinyorwa, mifananidzo, odhiyo, vhidhiyo, kana kodhi.

Mutauro Mukuru (LLM)

Mutauro wemodhi yakadzidziswa pane yakakura text corpora kugadzira nekuongorora zvinyorwa.

Multimodal Model

Modhi inogona kugadzirisa kana kugadzira akawanda emhando dzedata senge zvinyorwa, mufananidzo, uye odhiyo.

Mutauro-agnostic tokenizer inodzidza subword mauniti zvakananga kubva kune yakaomeswa zvinyorwa pasina pre-kupatsanurwa pachena.

Mucherechedzo wegiredhi-yechikoro masvomhu emazwi matambudziko anoshandiswa kuongorora nhanho-ne-nhanho kufunga mumienzaniso yemitauro.

Maitirwo ekupatsanura mavara kuita tokens yemhando yekuisa.

Dzidza zvakawanda mumadhairekitori edu emahara

SentencePiece Tokenization WordPiece Tokenization Tokenizer-Free Byte-Level Models Speculative Streaming and Multi-Token Prediction

Onawo

Temperature System Prompt Tool Use Synthetic Data Top-k Sampling Supervised Learning Top-p (Nucleus) Sampling Sparse Model

Bhurawuza yakazara AI Glossary Ongorora ese emahara AI madhairekitori