Ini Kutenderera Audio Codec
Mimi ndeye neural odhiyo codec inomanikidza kutaura kuita diki diki diki tokeni munguva chaiyo, saka maAI mamodheru anogona kuteerera uye kutaura neakadzika kwazvo latency.
Overview
Mimi ndeye neural odhiyo codec inomanikidza kutaura kuita diki diki diki tokeni munguva chaiyo, saka maAI mamodheru anogona kuteerera uye kutaura neakadzika kwazvo latency. Ndiyo inonzwika musana kuseri kweKyutai's Moshi voice modhi.
Ini Kutenderera Audio Codec inogara muodhiyo-AI workflows inoshandura kutaura, mimhanzi, uye ruzha rwekutaurirana, kuwanikwa, uye kugadzirwa kwenhau.
Deep Dive
Mimi, akaburitswa neFrench lab Kyutai muna 2024, ndeye neural codec inoshandura 24 kHz odhiyo kuita rukova rwe discrete tokens paanenge 1.1 kbps uye chete 12.5 tokens pasekondi. Inoshandisa encoder-decoder ine yasara vector quantization (RVQ), kupatsanura tokeni kuita 'semantic' yekutanga nhanho yakanyungudutswa kubva kune inozvitarisisa yekutaura modhi (WavLM) pamwe akati wandei 'acoustic' mazinga anobata manyorerwo ezwi. Kunyanya iyo iri kutenderera yakazara uye inokonzeresa: inoburitsa tokeni sezvo odhiyo ichisvika pane kumirira yakazara clip, ine angangoita 80 ms ye latency. Izvi zvinoita kuti modhi yemutauro ibate kutaura senge tokens, zvichiita kuti Moshi akwanise kutaura mune duplex yakazara uku achichengeta odhiyo yakagadziridzwa inonzwisisika uye yakasikwa.
Technical Insight
Hunyengeri hwaMimi ndeyekupatsanura-RVQ chirongwa. Iro bhuku rekutanga rekodhi rakadzidziswa nekurasikirwa kwe distillation kuenzanisa embeddings kubva kuWavLM, ichimanikidza kuti itakure phonetic 'zvinoreva,' nepo parallel acoustic codebooks inovakazve waveform ruzivo. A Transformer inoshanda mukati mebhodhoro, uye adversarial (GAN) kurasikirwa pane decoder inorodza kunaka kwekubuda. Causal convolutions inochengeta zvese zvichitenderera, saka latency inogara padyo ne80 ms.
Mastering Mimi Streaming Audio Codec
Kuvaka kunzwisisa kwakadzama, bata Mimi Kutenderera Audio Codec semuenzaniso wekushandisa, kwete chinhu chimwe chete. Tsanangura zvaunoda, jekesa fungidziro, uye patsanura izvo zvingaitwe nehurongwa nekuvimbika kubva kune zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa Mimi Streaming Audio Codec zvinobata mhando, latency, uye mvumo sezvikamu zvakakosha zvakaenzana zvehurongwa hwekuendesa. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza. Panguva imwecheteyo, kusashandiswa kweIzwi zvisizvo uye njodzi dzekuedzesera dzinowedzera kana chibvumirano chisipo. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza.
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Zvikwata zveMedia zvinogona kutumira odhiyo yakakwenenzverwa nekukurumidza nemabhajeti madiki.
Zvikwata zveMedia zvinogona kutumira odhiyo yakakwenenzverwa nekukurumidza nemabhajeti madiki. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Masisitimu anotarisana nevatengi anogona kugadzirisa kutaurirana kwekutaura pamwero mukuru.
Masisitimu anotarisana nevatengi anogona kugadzirisa kutaurirana kwekutaura pamwero mukuru. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
Kusimbisa Kyutai's Moshi yakazara-duplex izwi mubatsiri kuti ikwanise kuteerera uye kutaura panguva imwe chete
Kuburitsa ma tokeni ekutaura mumhando yemutauro yeshanduro yenguva chaiyo yekutaura-kukutaura
Mazwi emafoni epamusoro-pasi-bitrate (~ 1.1 kbps) yemamiriro etiweki asina kunaka kana akazara
Kuratidza odhiyo yekutaura kunogadzira uye mameseji-ku-kutaura-mapaipi anokonzeresa pamusoro pekurira semavara
Maitiro Ekuita
Ini Kutenderera Audio Codec mukuita
Kusimbisa Kyutai's Moshi yakazara-duplex izwi mubatsiri kuti ikwanise kuteerera uye kutaura panguva imwe chete.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Ini Kutenderera Audio Codec mukuita
Kuburitsa ma tokeni ekutaura mumhando yemutauro yeshanduro yenguva chaiyo yekutaura-kukutaura.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Ini Kutenderera Audio Codec mukuita
Mazwi emafoni epamusoro-pasi-bitrate (~ 1.1 kbps) yemamiriro etiweki asina kunaka kana akazara.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Ini Kutenderera Audio Codec mukuita
Kuratidza odhiyo yekutaura kunogadzira uye mameseji-ku-kutaura-mapaipi anokonzeresa pamusoro pekurira semavara.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Kushandisa izwi zvisizvo uye njodzi dzekuedzesera dzinowedzera kana chibvumirano chisipo.
Kururama kunogona kudonha mumitauro, mataurirwo, kana nharaunda dzine ruzha.
Synthetic audio inogona kukanganisa kutaura kwechokwadi isina mavara akajeka.
Implementation Roadmap
Wana mvumo yakajeka yekutora inzwi, kugadzira, uye kushandisa zvakare.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Yedza mhando pavatauri vakasiyana uye mamiriro ekumashure.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Tsanangura apo munhu anofanira kuongorora kana kubvumidza zvabuda.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Label synthetic odhiyo uye chengetedza marekodhi ekuzvidavirira.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Ramba Uchiongorora
Check your understanding
Test yourself: take the Mimi Streaming Audio Codec quiz