What is Tokenization? Definition | AI Understanding

Incazelo

Inqubo yokuhlukanisa umbhalo ube amathokheni okufaka imodeli.

Imigomo ehlobene

Iwindi Lokuqukethwe

Inani eliphakeme lamathokheni okufakwayo imodeli yolimi engawacubungula ngesikhathi esisodwa.

I-BPE (Byte Pair Encoding)

I-algorithm yethokheni yegama elingaphansi ehlanganisa ukupheya kwezinhlamvu ezivame kakhulu zibe amathokheni asebenziseka kabusha.

Ingxenye yemodeli eguqula okokufaka kube izethulo ezifihlekile.

Imodeli ye-Multimodal

Imodeli engacubungula noma ikhiqize izinhlobo eziningi zedatha njengombhalo, isithombe, nomsindo.

I-OCR (Ukubonwa Kwezinhlamvu Ezikhanyayo)

Ubuchwepheshe obuguqula umbhalo ezithombeni noma izikena zibe umbhalo ofundeka ngomshini.

Ithokheni ye-language-agnostic efunda amayunithi egama elingaphansi ngokuqondile embhalweni ongahluziwe ngaphandle kokuhlukanisa kusengaphambili endaweni emhlophe.

Funda kabanzi kumhlahlandlela wethu wamahhala

SentencePiece Tokenization WordPiece Tokenization Tokenization Subword Tokenization

Bhekafuthi

Token Tool Use Temperature Top-k Sampling System Prompt Top-p (Nucleus) Sampling Synthetic Data Transfer Learning

Phequlula Uhla lwamagama olugcwele lwe-AI Hlola yonke imihlahlandlela ye-AI yamahhala