SoundStream Neural Codec
SoundStream ndeye Google's end-to-end neural audio codec inodzvanya kutaura nemimhanzi kune yakaderera zvakanyanya bitrate uchichengetedza mhando.
Overview
SoundStream ndeye Google's end-to-end neural audio codec inodzvanya kutaura nemimhanzi kune yakaderera zvakanyanya bitrate uchichengetedza mhando. Izvo zvine basa nekuti inorova echinyakare macodecs seOpus panguva imwe chete bitrate uye inopa masimba emazuva ano ekugadzira odhiyo modhi.
SoundStream Neural Codec inogara muodhiyo-AI workflows inoshandura kutaura, mimhanzi, uye ruzha rwekutaurirana, kuwanikwa, uye kugadzirwa kwenhau.
Deep Dive
Yakaunzwa ne Google muna 2021, SoundStream ndeye yakazara neural codec yakavakwa kubva pazvidimbu zvitatu zvakadzidziswa pamwechete: convolutional encoder inoshandura mbishi waveform kuita compact kutevedzana kwemavector, inosara vector quantizer (RVQ) iyo inosiyanisa iyo inovandudza mavheji uye inogadzirisa mavheji. Iyo inodzidziswa nekurasikirwa kwekuvaka patsva uye GAN-maitiro anopikisa anosarura, saka zvinobuda zvinonzwika sechisikigo kwete kungovhara nhamba. Chimiro chakamira chiri 'scalable' kana quantizer-dropout kudzidziswa: modhi imwe chete inogona kushanda pamabitrate kubva panosvika 3 kusvika 18 kbps nekushandisa akawanda kana mashoma quantizer layer pakufungidzira, pasina kudzidziswazve. Pa3 kbps inonzi inodarika Opus pa12 kbps mukuteerera bvunzo, kubata kutaura, mimhanzi, uye zvakajairika odhiyo mune imwe modhi inogona kumhanya munguva chaiyo pa smartphone CPU.
Technical Insight
Iyo waveform inopfuura nemakonduru ane strided ayo anodzika sample zvakanyanya, achigadzira imwe emedding pafuremu (e.g. 75 mafuremu/sekondi). RVQ inozoisa encode ega ega senge murwi wemakodhi ebhuku indices. Bitrate yakaenzana neframe rate nguva nhamba yeanoshanda quantizers nguva mabhiti pacodebook. Quantizer dropout inodzora zvisina tsarukano murwi weRVQ panguva yekudzidziswa, ichimanikidza ekare macodebook kutakura ruzivo rwakanyanya kukosha kuitira kuti codec ideredze zvakanaka pamitengo yakaderera.
Mastering SoundStream Neural Codec
Kuvaka kunzwisisa kwakadzama, bata SoundStream Neural Codec semuenzaniso wekushandisa, kwete chinhu chimwe chete. Tsanangura zvaunoda, jekesa fungidziro, uye patsanura izvo zvingaitwe nehurongwa nekuvimbika kubva kune zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa SoundStream Neural Codec zvinobata mhando, latency, uye mvumo sezvikamu zvakakosha zvakaenzana zvehurongwa hwekuendesa. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza. Panguva imwecheteyo, kusashandiswa kweIzwi zvisizvo uye njodzi dzekuedzesera dzinowedzera kana chibvumirano chisipo. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza.
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Zvikwata zveMedia zvinogona kutumira odhiyo yakakwenenzverwa nekukurumidza nemabhajeti madiki.
Zvikwata zveMedia zvinogona kutumira odhiyo yakakwenenzverwa nekukurumidza nemabhajeti madiki. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Masisitimu anotarisana nevatengi anogona kugadzirisa kutaurirana kwekutaura pamwero mukuru.
Masisitimu anotarisana nevatengi anogona kugadzirisa kutaurirana kwekutaura pamwero mukuru. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
Kudzvanya manzwi ekufona kusvika ~ 3 kbps uku ichinzwika kujeka kupfuura legacy codecs pakakwirira bitrate
Kugadzira ma tokeni ekuteerera anodyisa Google's AudioLM neMusicLM anogadzirwa modhi.
Real-time low-bandwidth audio kutenderera pane nharembozha ine pa-CPU encoding uye decoding
Kuchengeta kana kufambisa mimhanzi uye ambient ruzha zvinobudirira mune imwechete modhi inobata marudzi ese emukati
Maitiro Ekuita
SoundStream Neural Codec mukuita
Kudzvanya manzwi ekufona kusvika ~ 3 kbps uku ichinzwika kujeka kupfuura legacy codecs pakakwirira bitrate.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
SoundStream Neural Codec mukuita
Kugadzira maodhiyo tokeni anodyisa Google's AudioLM neMusicLM anogadzirwa modhi.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
SoundStream Neural Codec mukuita
Real-time low-bandwidth audio kutenderera pane nharembozha ine pa-CPU encoding uye decoding.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
SoundStream Neural Codec mukuita
Kuchengeta kana kufambisa mimhanzi uye ambient ruzha zvinobudirira mune imwechete modhi inobata marudzi ese emukati.
Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Kushandisa izwi zvisizvo uye njodzi dzekuedzesera dzinowedzera kana chibvumirano chisipo.
Kururama kunogona kudonha mumitauro, mataurirwo, kana nharaunda dzine ruzha.
Synthetic audio inogona kukanganisa kutaura kwechokwadi isina mavara akajeka.
Implementation Roadmap
Wana mvumo yakajeka yekutora inzwi, kugadzira, uye kushandisa zvakare.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Yedza mhando pavatauri vakasiyana uye mamiriro ekumashure.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Tsanangura apo munhu anofanira kuongorora kana kubvumidza zvabuda.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Label synthetic odhiyo uye chengetedza marekodhi ekuzvidavirira.
Bata izvi segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gap, uye wobva wawedzera kushandiswa.
Ramba Uchiongorora
Check your understanding
Test yourself: take the SoundStream Neural Codec quiz