What is Token? Definition | AI Understanding

Itumọ

Apapọ ọrọ ti a ṣe nipasẹ awọn awoṣe ede, gẹgẹbi ọrọ ege tabi aami.

Awọn ofin ti o jọmọ

AI Aṣẹ̀dá

Awọn eto AI ti o gbejade akoonu titun gẹgẹbi ọrọ, awọn aworan, ohun, fidio, tabi koodu.

Awoṣe Ede nla (LLM)

Awoṣe ede ti a ṣe ikẹkọ lori titobi ọrọ corpora lati ṣe ipilẹṣẹ ati itupalẹ ọrọ.

Multimodal Awoṣe

Awoṣe ti o le ṣe ilana tabi ṣe ipilẹṣẹ awọn oriṣi data lọpọlọpọ gẹgẹbi ọrọ, aworan, ati ohun.

GbólóhùnPiece

Tokenizer ede-agnostic ti o kọ ẹkọ awọn ipin ọrọ-ọrọ taara lati ọrọ aise laisi pipin-tẹlẹ lori aaye funfun.

Aṣepari ti awọn iṣoro ọrọ mathimatiki ite-ile-iwe ti a lo lati ṣe iṣiro ero-igbesẹ-igbesẹ ni awọn awoṣe ede.

Ilana ti pipin ọrọ sinu awọn ami fun titẹ sii awoṣe.

Kọ ẹkọ diẹ sii ninu awọn itọsọna ọfẹ wa

SentencePiece Tokenization WordPiece Tokenization Tokenizer-Free Byte-Level Models Speculative Streaming and Multi-Token Prediction

Wo tun

Temperature System Prompt Tool Use Synthetic Data Top-k Sampling Supervised Learning Top-p (Nucleus) Sampling Sparse Model

Ṣawakiri ni kikun AI Gilosari Ṣawari gbogbo awọn itọsọna AI ọfẹ