Ikibazo gihuza ibibazo · Hagati Urwego

Tokenisation na Byte Pair Encoding Ikibazo

Ikizamini cyerekana indangamuntu, inshuro nyinshi zishingiye kuri BPE guhuza, byte-urwego rwo gukwirakwiza, guhuza gahunda, n'impamvu kubara ibimenyetso bitandukanye no kubara ijambo.

Inzira zijyanye no kuyoboraTokenisation na Byte Byombi Encoding
Ikibazo 1 Bya 8

Niki tokenisation itanga urugero rwururimi rwo gusoma?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Niki tokenisation itanga urugero rwururimi rwo gusoma?
    • Tokens, nindangamuntu yuzuye yashizwe mubice byinyandiko
    • Amajwi mabi
    • Imashini ya pixel
    • Gukusanya kode yimashini
  2. Nigute Byte Pair Encoding yubaka amagambo yayo?
    • Mugenera ikimenyetso kimwe kumagambo yose mu nkoranyamagambo
    • Muguhuza inshuro nyinshi guhuza ibimenyetso byinshi byegeranye
    • Mugihe cyo gutondekanya inyuguti
    • Muguhindura inyandiko murundi rurimi
  3. Ni ikihe kibazo uburyo bwa subword uburyo nka BPE bukemura ugereranije nijambo-urwego tokenisation?
    • Bakora buri nteruro ikimenyetso kimwe
    • Bakuraho ibikenewe byamagambo ayo ari yo yose
    • Bakoresha amagambo adasanzwe cyangwa atagaragara bayagabanyamo ibice bikoreshwa
    • Bakorera icyongereza gusa
  4. Ni izihe nyungu za byte-urwego BPE, nkuko bikoreshwa muri moderi ya GPT?
    • Ikoresha ibimenyetso bike kurenza ubundi buryo bwindimi zose
    • Bikuraho gukenera guhugura icyitegererezo
    • Irimo gusa inyuguti zicyongereza
    • Irashobora kwerekana inyandiko iyo ari yo yose ya Unicode, harimo emoji n'ururimi urwo arirwo rwose, nta kunanirwa-amagambo
  5. Ni ukubera iki icyitegererezo cyerekana ibimenyetso akenshi gitandukanya numubare wamagambo mu nteruro?
    • Umwanya, inyuguti nkuru, n'amagambo adasanzwe ahindura uburyo inyandiko igabanijwe, kandi ijambo rimwe rishobora guhinduka ibimenyetso byinshi
    • Tokens ibarwa mururimi rudasanzwe
    • Buri jambo buri gihe ni ibimenyetso bibiri
    • Ibara ryaciwe biterwa gusa no kuruhuka
  6. Ni uwuhe murima Byte Pair Encoding algorithm yatijwe mbere?
    • Gutanga amashusho
    • Guhagarika amakuru
    • Kwerekana Ububikoshingiro
    • Cryptography
  7. After BPE training creates its vocabulary, what does an encoder use to tokenize new text?
    • A new merge ranking learned separately for each sentence
    • A dictionary of whole words only
    • The learned vocabulary and ordered merge rules applied to the input symbols
    • A translation table that changes the text language
  8. Why can byte-level BPE encode a character it never saw as a whole token during training?
    • The tokenizer invents new model parameters for each character
    • It drops every unseen character
    • The model automatically translates it to English first
    • The UTF-8 input can be represented as a sequence of bytes from a finite base alphabet