VALL-E ak xeetu làkk Codec
VALL-E dafa soppi bind-ci-kaddu nekk jafe-jafe modeling làkk ci token codec audio, loolu mooy tax baat bi mëna klone ci ñetti segond ci misaal bi.
Résumé
It showed that the same next-token prediction powering text LLMs can generate remarkably natural, expressive speech.
Plongeur bu xóot
Microsoft moo ko siiwal ci ndoortelu 2023, VALL-E dafay jàppee waxtaan wi ni modelu làkk. Du wax luy waaja am ci spectrogram, waaye dafay wax luy waaja am ci token akustik yu diskret yi ci codec neural (EnCodec), suko defee generation nekk token bi ci topp di wax luy waaja am ci kaw vocabulaire audio. Suñu joxee enregistrement bu 3 seconde bu benn waxkat buñu gisul ak mbind biñ bëgga bind, VALL-E dafay wéy ci baat biy waxkat bi, di baña yàq timbre bi ba ci environmaa akustik bi. Ñu ngi ko tàggat ci lu tollu ci 60 000 waxtuy wax, muy lu ëpp bu baax li ñuy woowe TTS, loolu moo ko may cloning zero-shot bu dëgër. Ndax token codec yi dañuy layered (jaaraleko ci RVQ), VALL-E dafay jëfandikoo ñaari etap: benn model autoregressive dafay wax lu njëkk, stream token bu dëgër biñ kondisione ci laaj bi, ak benn model bu amul autoregressive buy fees dessitu token yi. Bii rëset codec-LM inspiré na ñu ci topp lu melni VALL-E 2 ak yeneen xeetu fondation kàddu.
Gis-gis xarala
Li am solo mooy dekodage hybrid ci kaw jeton codec hierarchique. Etape autorégressive bi dafay wax luy am solo ci token codebook bu njëkk bi benn-benn, di jàpp prosody ak ëmbiit li. Dessitu téere kode yi, yuy yokk ay detay akustik yu rafet, ñu ngi leen wax luy waaja am ci paralel ci xeetu model bu dul autorégressive buñ tëral ci stream bu njëkk bi ak ci prompt bu waxkat bi. Xaajale bii dafay tëye kalite bi ci di moytu njëgu defar token bu nekk ci anam wu toppalante, te jëfandikoo codec dafay tekki ni wax ak bind mën nañu leen modele ak benn masin transformateur.
njeextalu pexe
Dugg ak yegg
Dafay gëna yombal jëfandikoo gi jaaraleko ci transkripsioŋ, nettali ak interfaasu baat.
Njëgg ak budget
Ekipu mejaa yi mën nañu yónnee audio bu leer ci anam wu gëna gaaw te seen xaalis gëna néew.
Gaawaay ak yaatuwaay
Sistem yiy jàkkarloo ak kiliyaan bi mën nañu def waxtaan ci anam wu gëna yaatu.
Ëlëgu VALL-E ak xeetu làkk Codec
Royuwaayi làkk codec dañuy boole wax ak royuwaayi làkk yu mag, di wane sistem yuñ boole muy déglu, xalaat ak wax ci benn royuwaay. Xaarandil dayo bu gëna baax ak ay mbiri art yu néew, defar ay streaming ci jamono dëgg, ak di gëna mëna yor yëg-yëg ak stil. Benn klonaas bu am doole biy tax VALL-E am njariñ ci jëfandikoo ak dubbing itam dafay yee ci deepfake ak jafe-jafe nangu, kon watermarking, kaaraange baat-xool, ak guardrails politik ñu ngi nekk lu am solo ci ni ñuy jëfandikoo sistem yooyu.
Doxal ci àdduna dëgg
Klone ab baat ci ay segond yu néew ci audio ngir assistant yuñ personaalise wala jumtukaayi jëfandikoo ngir defaraat baat bu réer
Lokalise ak dubbing wideo ci yeneen làkk te baña bàyyi timbre bi waxkat bi njëkka am
Defar ay nettali yu fëgër, méngoo ak muy tekki, di baña yàq environmaa akustik bi ci enregistremaa bi
Dafay nekk yaxu ndigg ci kàddu yi ci assistant multimodal yiy dégg ak defar audio yuñ wax
Risk yi ak balustrade yi
Jëfandikoo baat ci anam wu jaarul yoon ak niru ak nit dafay gëna yokk sudee nanguwul.
Jaar-jaar mën na wàññeeku ci aksan yi, dialect yi wala barab yu bari xumbaay.
Audio synthetik mën nañu ko jaawale ak wax ju dëggu sudee amul etiket bu leer.
Roadmap ngir samp gi
Wutal ndigal bu leer ngir jàpp baat bi, klone ko ak jëfandikoowaat ko.
Saytu kalite ci kàddukat yu bari ak anam yu bari ci ginaaw.
Mandargal kañ la nit wara xoolaat wala nangu ay génne.
Etiketu audio synthetik te nga denc dokimaa ci fimu bawoo ngir mëna lim.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the VALL-E and Codec Language Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Mën-mën yu feeñ ci misaali làkk yu mag
Laaj yi ñuy faral di laaj
What is VALL-E and Codec Language Models?
VALL-E dafa soppi bind-ci-kaddu nekk jafe-jafe modeling làkk ci token codec audio, loolu mooy tax baat bi mëna klone ci ñetti segond ci misaal bi. Mu wane ni benn token bi ci topp biy wax luy waaja am ci LLMs mën na defar kàddu yu yéeme, yu fësal kàddu.
Lan mooy xalaat bu am solo bi wuutale VALL-E ak sistemu bind-ci-kaddu yu njëkk ya?
VALL-E dafay reframes TTS ni token bi ci topp ci kaw token codec audio yu diskret, di jëfandikoo defar kàddu ni xeetu làkk.
Ñaata audio la VALL-E soxla ngir klone baat buy wax bu bees?
VALL-E mën na def klonaasu baat bu zero-shot ci misaalu 3 sikon ci benn waxkat buñu gisul.
Ban codec neuronal la VALL-E di jëfandikoo ngir defar ay token audio?
VALL-E dafa tabax ci kaw EnCodec bu Meta, di wax luy waaja am ci ay token akustik yu wuute ngir boole kàddu yi.
Lan moo waral VALL-E di jëfandikoo etape buy dellu ginaaw ak bu dul dellu ginaaw?
Jeton codec yi dañuy hierarchique jaaraleko ci RVQ; VALL-E dafay wax luy waaja am ci dex gu njëkk gi autoregressively ak dessitu jetons yi ci paralel ngir am njariñ.
Lu tollu ci ñaata done ci kàddu lañu tàggatee VALL-E, loolu mooy tax ñu mëna def klonaas bu am doole?
VALL-E dañu ko tàggat ci lu tollu ci 60,000 waxtu wax, lu ëpp luñu daan jëfandikoo ci TTS, loolu moo gëna dooleel zero-shot generalisation.