quiz bu lëkkale ak tegtal · Diggante Niveau

Tokenisasioŋ ak quiz kodage ñaari byte

Dafay saytu ID token yi, boole BPE yi lalu ci fréquence, coverage byte-niveau, njuréefi boole yi, ak lu tax lim token yi wuute ak lim baat yi.

Yooni njiit yu jëm ciTokenisasioŋ ak kodage ñaari byte
Laaj 1 de 8

Lan la tokenization di defar ngir modelu làkk mëna jàng?

Xoolaatal laaj bu nekk8 laaj

Sooy def quiz bi, tontu yi ak leeral yi dañuy feeñ.

  1. Lan la tokenization di defar ngir modelu làkk mëna jàng?
    • Jetons, muy ID integer yuñ boole ci ay mbind
    • Formu onde audio yu ñor
    • Griy pixel
    • Kodu masin buñ dajale
  2. naka la kodage Byte Pair di tabaxee vocabulaire bi?
    • Soo joxee benn token ci baat bu nekk ci diksoneer bi
    • Ni ñuy boole ñaari màndarga yi gëna bari luñu leen di dendal
    • Ni ñuy boolee araf yi ci anam wu mucc
    • Sooy njëkka tekki mbind ci beneen làkk
  3. Ban jafe-jafe la pexe yu ndaw yu melni BPE di saafara suñu ko méngale ak tokenization ci wàllu baat?
    • Dañuy def frase bu nekk benn màndarga
    • Dañu dindi bépp soxla ci vocabulaire
    • Dañuy jëfandikoo kàddu yu bariwul wala yuñu gisul, ñu xaaj leen ay pàcc yuñ mëna jëfandikoowaat
    • Angale rek lañuy liggéey
  4. Lan mooy njariñu BPE niveau byte, ni ñu koy jëfandikoo ci model GPT yi?
    • Dafay jëfandikoo ay token yu néew yeneen xeeti jëfandikoo ci làkk yépp
    • Dina dindi soxla tàggat model bi
    • Dafay kodde arafu angale kese
    • Mën na màndargaal bépp mbindu Unicode, lu ci melni emoji ak bépp làkk, te amul benn njuumte ci vocabulaire bi
  5. Lan moo waral limu token yi ci benn model di wuute ak limu baat yi ci benn frase?
    • Espace yi, capitalisation yi, ak baat yu bari yi dañuy soppi ni mbind mi di xaajalee, te benn baat mën na nekk ay token yu bari
    • Jetons yi deñukoy lim ci làkk wu bari
    • Baatu bu nekk dafay am ñaari token
    • Token yi dañuy aju ci ponksioŋ yi
  6. Ci ban barab lañu njëkka leblee algorithmu enkodaasu ñaari byte yi?
    • Rendu nataal
    • Kompresioŋ done
    • Indexation de base de done
    • Kriptograafi
  7. After BPE training creates its vocabulary, what does an encoder use to tokenize new text?
    • A new merge ranking learned separately for each sentence
    • A dictionary of whole words only
    • The learned vocabulary and ordered merge rules applied to the input symbols
    • A translation table that changes the text language
  8. Why can byte-level BPE encode a character it never saw as a whole token during training?
    • The tokenizer invents new model parameters for each character
    • It drops every unseen character
    • The model automatically translates it to English first
    • The UTF-8 input can be represented as a sequence of bytes from a finite base alphabet