Baatu Glossaire IA

Lan mooy Tokenisasioŋ?

Maanaam

Xeetu xaaj mbind ci ay jeton ngir dugal ci model.

Xoolal Glossaire IA bi yéppSaytu bépp tegtal IA bu amul fayda

Term yu ci méngoo

Palanteer ëmbiitTolluwaayu dugal bi gëna mag bi benn xeetu làkk mëna def benn yoon.BPE (kodage ñaari byte)Algorithme tokenization subword buy boole ñaari araf yi gëna bari ci ay token yuñ mëna jëfandikoowaat.EnkodeurKoposaŋ bi ci xeetu model biy soppi dugal ci misaal yu nëbbu.Modèle multimodalModèle bu mëna defar wala defar ay xeeti done yu bari lu ci melni bind, nataal wala audio.OCR (Xamme arafu Optik)Xarala yuy soppi mbind ci nataal wala skane ci mbind mu masin mëna jàng.Piyeesu fraseTokenizer bu xamul làkk, di jàng ay subword ci bind bu ñor te duko njëkka xaaj ci espace bu weex.

Nekk leen ci sunuy tegtal yu amul fayda

SentencePiece TokenizationWordPiece TokenizationTokenizationSubword Tokenization

Xoolal itam

TokenTool UseTemperatureTop-k SamplingSystem PromptTop-p (Nucleus) SamplingSynthetic DataTransfer Learning