Overview
SoundStream ndeye Google's end-to-end neural audio codec inodzvanya kutaura nemimhanzi kune yakaderera zvakanyanya bitrate uchichengetedza mhando. Izvo zvine basa nekuti inorova echinyakare macodecs seOpus panguva imwe chete bitrate uye inopa masimba emazuva ano ekugadzira odhiyo modhi.
SoundStream Neural Codec inogara muodhiyo-AI workflows inoshandura kutaura, mimhanzi, uye ruzha rwekutaurirana, kuwanikwa, uye kugadzirwa kwenhau.
Deep Dive
Yakaunzwa ne Google muna 2021, SoundStream ndeye yakazara neural codec yakavakwa kubva pazvidimbu zvitatu zvakadzidziswa pamwechete: convolutional encoder inoshandura mbishi waveform kuita compact kutevedzana kwemavector, inosara vector quantizer (RVQ) iyo inosiyanisa iyo inovandudza mavheji uye inogadzirisa mavheji. Iyo inodzidziswa nekurasikirwa kwekuvaka patsva uye GAN-maitiro anopikisa anosarura, saka zvinobuda zvinonzwika sechisikigo kwete kungovhara nhamba. Chimiro chakamira chiri 'scalable' kana quantizer-dropout kudzidziswa: modhi imwe chete inogona kushanda pamabitrate kubva panosvika 3 kusvika 18 kbps nekushandisa akawanda kana mashoma quantizer layer pakufungidzira, pasina kudzidziswazve. Pa3 kbps inonzi inodarika Opus pa12 kbps mukuteerera bvunzo, kubata kutaura, mimhanzi, uye zvakajairika odhiyo mune imwe modhi inogona kumhanya munguva chaiyo pa smartphone CPU.
Technical Insight
Iyo waveform inopfuura nemakonduru ane strided ayo anodzika sample zvakanyanya, achigadzira imwe emedding pafuremu (e.g. 75 mafuremu/sekondi). RVQ inozoisa encode ega ega senge murwi wemakodhi ebhuku indices. Bitrate yakaenzana neframe rate nguva nhamba yeanoshanda quantizers nguva mabhiti pacodebook. Quantizer dropout inodzora zvisina tsarukano murwi weRVQ panguva yekudzidziswa, ichimanikidza ekare macodebook kutakura ruzivo rwakanyanya kukosha kuitira kuti codec ideredze zvakanaka pamitengo yakaderera.
Mastering SoundStream Neural Codec
SoundStream ndeye Google's end-to-end neural audio codec inodzvanya kutaura nemimhanzi kune yakaderera zvakanyanya bitrate uchichengetedza mhando. Izvo zvine basa nekuti inorova echinyakare macodecs seOpus panguva imwe chete bitrate uye inopa masimba emazuva ano ekugadzira odhiyo modhi. SoundStream Neural Codec inogara muodhiyo-AI workflows inoshandura kutaura, mimhanzi, uye ruzha rwekutaurirana, kuwanikwa, uye kugadzirwa kwenhau. Kuvaka kunzwisisa kwakadzama, bata SoundStream Neural Codec semuenzaniso wekushandisa, kwete chinhu chimwe chete: tsanangura zvinodikanwa, jekesa fungidziro, uye patsanura izvo zvingaitwe nehurongwa hwakavimbika kubva kune zvichiri kuda kutonga kwenyanzvi.
Mukuita, zvikwata zvakasimba zvinoshandisa SoundStream Neural Codec zvinobata mhando, latency, uye mvumo sezvikamu zvakakosha zvakaenzana zvehurongwa hwekuendesa. Ivo vanonyora zvakajeka maitiro ebudiriro, bvunzo vachipokana ne data rechokwadi uye mafambiro ebasa, uye iterate zvichibva pane zvakacherechedzwa maitiro ekutadza kwete kuhwina-nguva imwe chete yebhenji. Apa ndipo apo kunzwisisa kwe theoretical kunoshanduka kuve kugona kwakasimba pane chigadzirwa, mutemo, uye mashandiro.
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza. Panguva imwecheteyo, kusashandiswa kweIzwi zvisizvo uye njodzi dzekuedzesera dzinowedzera kana chibvumirano chisipo. Nzira yakatsiga ndeyekubatanidza kukurumidza kuyedza nekutonga: mhanyisa vatyairi vendege, tora humbowo, buritsa matanda esarudzo, uye urambe uchivandudza chengetedzo semaitiro emuenzaniso, zvinotarisirwa nemushandisi, uye zvinodikanwa zvekutonga.
Strategic Impact
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza.
Inonatsiridza kusvikika kuburikidza nekunyora, kurondedzera, uye mazwi ekubatanidza. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Zvikwata zveMedia zvinogona kutumira odhiyo yakakwenenzverwa nekukurumidza nemabhajeti madiki.
Zvikwata zveMedia zvinogona kutumira odhiyo yakakwenenzverwa nekukurumidza nemabhajeti madiki. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Masisitimu anotarisana nevatengi anogona kugadzirisa kutaurirana kwekutaura pamwero mukuru.
Masisitimu anotarisana nevatengi anogona kugadzirisa kutaurirana kwekutaura pamwero mukuru. Mukutumirwa kwemhando yepamusoro, izvi zvinoshandurirwa kuita mitemo inoyerwa yekushanda, miganhu yevaridzi, uye tsika dzekudzokorora dzinodzokororwa kuitira kuti zvikwata zvikwire kuvimba pane kukwidza kusajeka.
Real-World Implementation
Kudzvanya manzwi ekufona kusvika ~ 3 kbps uku ichinzwika kujeka kupfuura legacy codecs pakakwirira bitrate
Kugadzira ma tokeni ekuteerera anodyisa Google's AudioLM neMusicLM anogadzirwa modhi.
Real-time low-bandwidth audio kutenderera pane nharembozha ine pa-CPU encoding uye decoding
Kuchengeta kana kufambisa mimhanzi uye ambient ruzha zvinobudirira mune imwechete modhi inobata marudzi ese emukati
Maitiro Ekuita
SoundStream Neural Codec mukuita
Kudzvanya manzwi ekufona kusvika ~ 3 kbps uku ichinzwika kujeka kupfuura legacy codecs pakakwirira bitrate.
Kudzvanya manzwi ekudanwa ku ~ 3 kbps uku ichinzwika kujeka kupfuura legacy codecs pakakwirira bitrate Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura hunhu hwepamberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
SoundStream Neural Codec mukuita
Kugadzira maodhiyo tokeni anodyisa Google's AudioLM neMusicLM anogadzirwa modhi.
Kugadzira madhidhiyo tokeni anodyisa Google's AudioLM neMusicLM anogadziridza modhi Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura mhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi emupendero, uye kuteedzera zvese zvakawanikwa zvechigadzirwa uye mutengo wekukanganisa nekufamba kwenguva.
SoundStream Neural Codec mukuita
Real-time low-bandwidth audio kutenderera pane nharembozha ine pa-CPU encoding uye decoding.
Real-time low-bandwidth audio kutenderera pane nharembozha ine pa-CPU encoding uye decoding Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
SoundStream Neural Codec mukuita
Kuchengeta kana kufambisa mimhanzi uye ambient ruzha zvinobudirira mune imwechete modhi inobata marudzi ese emukati.
Kuchengeta kana kufambisa mimhanzi uye ambient ruzha zvinobudirira mune imwe modhi inobata marudzi ese emukati Matimu anowanzo kuwana mhedzisiro iri nani kana achinge atsanangura emhando yepamusoro kumberi, chengetedza nzira yekukwira kwevanhu yemakesi ekumucheto, uye kuteedzera zvese zvakawanikwa zvechigadzirwa nemitengo yekukanganisa nekufamba kwenguva.
Njodzi & Guardrails
Kusashandisa izwi zvisizvo uye njodzi dzekuedzesera dzinowedzera kana chibvumirano chisipo.
Kururama kunogona kudonha mumitauro, mataurirwo, kana nharaunda dzine ruzha.
Synthetic audio inogona kukanganisa kutaura kwechokwadi isina mavara akajeka.
Implementation Roadmap
Wana mvumo yakajeka yekutora inzwi, kugadzira, uye kushandisa zvakare.
Wana mvumo yakajeka yekutora inzwi, kugadzira, uye kushandisa zvakare. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Yedza mhando pavatauri vakasiyana uye mamiriro ekumashure.
Yedza mhando pavatauri vakasiyana uye mamiriro ekumashure. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Tsanangura apo munhu anofanira kuongorora kana kubvumidza zvabuda.
Tsanangura apo munhu anofanira kuongorora kana kubvumidza zvabuda. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.
Label synthetic odhiyo uye chengetedza marekodhi ekuzvidavirira.
Label synthetic odhiyo uye chengetedza marekodhi ekuzvidavirira. Bata nhanho yega yega segedhi rehumbowo: kana maitiro asina kusangana, imbomira kuburitsa, vhara gaka, uye wobva wawedzera kushandiswa.