Luy BPE (kodage ñaari byte)? Tekki | AI Understanding

Maanaam

Algorithme tokenization subword buy boole ñaari araf yi gëna bari ci ay token yuñ mëna jëfandikoowaat.

Term yu ci méngoo

Xeetu xaaj mbind ci ay jeton ngir dugal ci model.

OCR (Xamme arafu Optik)

Xarala yuy soppi mbind ci nataal wala skane ci mbind mu masin mëna jàng.

Top-k jël misaal

Pexem dekodaas buy jël misaal ci k token yi ci topp.

Tasaaro ci ginaaw

Algorithm bi gëna am solo biy yeesal diisaayu model bi ci tasaare njuumti yiñ wax luy waaja am ci reso bi.

Modèle buy kode laajte yi ak këyit yi ci vecteur yu wuute suko defee ñu mëna leen méngale ci diir bu gàtt ci eskaal bi.

Algorithme decodage buy tëye limu kàndidaa yi ci kaw ci jéego bu nekk ngir am njariñ yu gëna am wërsëg.

Nekk leen ci sunuy tegtal yu amul fayda

ByteDance Doubao CycleGAN Unpaired Translation Tokenizer-Free Byte-Level Models Instant-NGP Hash Encoding

Xoolal itam

Sliding Window Attention SentencePiece ALiBi ANN (Approximate Nearest Neighbor)RoPE (Rotary Positional Embeddings)HNSW Positional Encoding Reranker

Xoolal Glossaire IA bi yépp Saytu bépp tegtal IA bu amul fayda