What is Tokenization? Definition | AI Understanding

Nkọwa

Usoro nke ikewa ederede ka ọ bụrụ akara maka ntinye ihe nlereanya.

Okwu ndị metụtara

Onodu ihe omimi

Ọnụ ego ntinye ntinye kacha nke ụdị asụsụ nwere ike hazie otu oge.

BPE (Nkọnye ụzọ abụọ Byte)

Algọridim n'ime okwu obere okwu na-ejikọta ụzọ abụọ agwakarị n'ime akara ngosi enwere ike ijikwa ya.

Akụkụ nke ihe nlereanya nke na-agbanwe ntinye ka ọ bụrụ ihe nnọchianya.

Multimodal Model

Ụdị nke nwere ike ịhazi ma ọ bụ mepụta ọtụtụ ụdị data dịka ederede, onyonyo, na ọdịyo.

OCR (Nchọpụta agwa anya)

Teknụzụ na-atụgharị ederede na onyonyo ma ọ bụ nyocha ka ọ bụrụ ederede igwe nwere ike ịgụ.

Asụsụ-agnostic tokenizer na-amụta nkeji obere okwu ozugbo site na ederede ederede na-enweghị tupu nkewa na oghere ọcha.

Mụtakwuo na ntuziaka efu anyị

SentencePiece Tokenization WordPiece Tokenization Tokenization Subword Tokenization

Hụkwa

Token Tool Use Temperature Top-k Sampling System Prompt Top-p (Nucleus) Sampling Synthetic Data Transfer Learning

Chọgharịa akwụkwọ nkọwa AI zuru ezu Nyochaa ntuziaka AI niile efu