VQ-VAE ak Latent yu wuute
VQ-VAE dafay tënk nataal yi, audio yi wala wideo yi ci benn griy bu ndaw bu am kod yu wuute yuñ jëlee ci téere kode buñ jàng, ci barabu nimero yu wéy.
Résumé
This discrete bottleneck lets powerful sequence models like Transformers treat media as 'tokens', much like words.
Plongeur bu xóot
VQ-VAE (Vecteur encodeur variationnel), van den Oord ak ay naataangoom ci DeepMind ñoo ko dugal ci 2017, encodeur la bu am bayaal bu nëbbu. Encoder dafay soppi nataal bi mu nekk griy bu am vecteur yu wéy; vecteur bu nekk dañu koy boole ci duggam bi gëna jege ci téere kode buñ jàng ci embeddings (quantisation vecteur). Decodeur bi dafay defaraat nataal bi ci kode kantise yooyu. Ginaaw latent yi leegi vocabulaire bu am àpp lañu ci indices yi, model bu wuute mën na jàng seen distribution ba noppi defar ëmbiit bu bees. Bii rëset bu am ñaari etap dafay dooleel DALL-E 1, Jukebox ngir music, ak VQGAN, loolu dafay yokk ñàkka xam ak xeex ngir tabaxaat yu gëna ñaw. VQ-VAE-2 dafa boole resolusioŋ yu bari ngir génne nataal yu dëggu.
Gis-gis xarala
Jéego kantite (argmin seet dëkkandoo bi gëna jege) mënul wuutale, moo tax VQ-VAE dafay jëfandikoo estimatër bu jub: gradient yi dañu leen kopie ci duggal dekodeer bi dellu ci génnug enkodeer bi melni kantite bi mooy dàntite bi. Taggat dafay boole ñàkka tabaxaat, ñàkka am codebook di xëcc embeddings yi ci encoder biy génn, ak ñàkka def encoder bi di tëye kode yi mu tànn. Benn ci jafe-jafe yiy faral di am mooy codebook bi dafay daanu, ndax kode yu néew rek lañuy jëfandikoo.
njeextalu pexe
Gaawaay ak yaatuwaay
Visual IA mën na otomatise saytu, gis ak etiketu liggéey ci eskaal.
Tabax tànneef
Ekipu kreatif yi mën nañu defar konsept yu gëna gaaw te duñu def lu bari ci loxo.
Ekip ak def liggéey
Liggéeyukaay yi mën nañu jëfandikoo siñaal nataal wala wideo yu jafewoon lool ci liggéey.
Ëlëgu VQ-VAE ak Latent yu wuute
latents diskret yi ñoo gëna am solo ci push bi ci model multimodal yuñ boole ngir tokenize nataal yi, audio yi ak wideo yi ci benn vocabulaire ak bind. Ay yokkute yu melni kantite scalar residual ak fini, téere kode yu gëna mag, ak jëfandikoo bu gëna baax ci ekilibre dañuy wàññi daanu gi ak gëna yokk njub. Bi xeetu yi di jéema xam ak defar ci anam yu bari, tokenizers yu dëgër yuñ tabax ci xalaati VQ-VAE dina ñu des ingredient fondamentaal, di gëna xëcc ak boole ak jegewaale diffusion latent yuy wéy.
Doxal ci àdduna dëgg
DALL-E 1 dafa jëfandikoo benn tokeniseer VQ-VAE bu wuute ngir Transformatër bi mëna defar ay nataal yu melni limu téere kode yi.
VQGAN boole VQ-VAE ak perte adversarial ak perceptuel ngir génne ay token nataal yu fëgër, yu am dayo bu kawe ngir defar art.
Jukebox OpenAI dafa jëfandikoo VQ-VAE ci audio bu ñor, di dajale music bi ci ay kod yu wuute ngir defar ay model.
VQ-VAE-2 dafa dajale ay latents yu wuute ngir dajale ay nataal yu bari te wóor yuy xëccoo ak GAN yu jamonoom.
Risk yi ak balustrade yi
Yelleefi nataal ak nangu mën na nekk risku yoon sudee fi ñu bawoo leerul.
Performance model bi mën na wuute ci leeraay bi, demographie bi ak environmaa bi.
Njuumteg positive yi mën nañu dem te kenn duko seetlu fileek xool wuñu buntu wóolu sa bopp.
Roadmap ngir samp gi
Mandargal kritërium nangug njub, woowaat ak njëgu njuumte.
Saytu ak done yu méngoo ak anam yi ñuy liggéeyee dëgg.
Yokk jàngat nit ngir xam fu wóorul dara wala am njeexital yu rëy.
Toppal model drift bi nga baaxal ko ginaaw bi kamera bi wala done yi soppeekoo.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the VQ-VAE and Discrete Latents quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Njaxas bu nëbbu ak boole nataal
Laaj yi ñuy faral di laaj
What is VQ-VAE and Discrete Latents?
VQ-VAE dafay tënk nataal yi, audio yi wala wideo yi ci benn griy bu ndaw bu am kod yu wuute yuñ jëlee ci téere kode buñ jàng, ci barabu nimero yu wéy. Bottleneck bu nëbbu bi dafay may model yu am doole yu melni Transformers ñu jàppee media yi ni ay 'token', lu melni kàddu.
Lan moo wuutale espace latente bu VQ-VAE ak VAE buñ miin?
VQ-VAE dafay wecci ay latente yu wéy ak ay kod yu wuute yuñ jëlee ci téere kod buñ jàng jaaraleko ci kantite vecteur.
naka la VQ-VAE di jaar ci gradient yi ci jéego kantite bu mënul wuute?
Estimatëru jaar-jaar bu jub bi dafay kopie gradient bi ci duggal dekodeer bi ci gennup enkodeer bi, muy jëfandikoo kantite bi ni dàntite ngir jàll bi ci ginaaw.
Luy 'codebook daanu'?
Codebook collapse dafay am sudee model bi dafa wéeru ci kode yu néew, yàq li ëpp ci codebook bi ba noppi gaañ wuute gi.
Lan moo waral latente yu wuute yi am njariñ ci modeling generatif ak Transformer yi?
Index yu diskret yi dañuy forme vocabulaire bu am àpp, kon model yu toppalante yu melni Transformers mën nañu jàng ak misaal seen séddale ni baat yi.
Lan la VQGAN yokk ci kaw rëset bu njëkk bi ci VQ-VAE?
VQGAN dafay yokk VQ-VAE ak benn diskriminatër ak ñàkka xam, di joxe ay token yu gëna fës, yu gëna leer.