What is Tokenization? Definition | AI Understanding

Ma'anarsa

Tsarin rarraba rubutu zuwa alamu don shigarwar ƙira.

Sharuɗɗa masu alaƙa

Matsakaicin adadin shigarwar alamun da samfurin harshe zai iya aiwatarwa a lokaci ɗaya.

BPE (Rubutun Rubutun Biyu)

Algorithm ɗin alamar ƙaƙƙarfan kalmomi wanda ke haɗa nau'ikan haruffa akai-akai zuwa alamun sake amfani da su.

Bangaren ƙirar da ke canza shigarwa zuwa wakilcin ɓoye.

Multimodal Model

Samfurin da zai iya sarrafawa ko samar da nau'ikan bayanai da yawa kamar rubutu, hoto, da sauti.

OCR (Gane Halayen Na gani)

Fasahar da ke canza rubutu a cikin hotuna ko dubawa zuwa rubutu mai iya karanta na'ura.

Alamar harshe-agnostic wanda ke koyan raka'a ƙananan kalmomi kai tsaye daga ɗan rubutu ba tare da an riga an raba kan farar fata ba.

Ƙara koyo a cikin jagororinmu na kyauta

SentencePiece Tokenization WordPiece Tokenization Tokenization Subword Tokenization

Duba kuma

Token Tool Use Temperature Top-k Sampling System Prompt Top-p (Nucleus) Sampling Synthetic Data Transfer Learning

Nemo cikakken AI ƙamus Bincika duk jagororin AI kyauta