音訊人工智慧指南

SoundStream 神經音訊編碼器

SoundStream is a neural audio codec from Google Research, published in July 2021, that compresses speech, music and general audio with a learned encoder, a residual vector quantizer and a learned decoder.

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Working out SoundStream's bitrate from its settings
  5. 現實世界的實施
  6. The Future of SoundStream Neural Audio Codec
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

In listening tests it matched or beat traditional codecs at a fraction of their bitrate, and its discrete codes became a standard way to turn audio into tokens for generative models.

重點摘要

  1. SoundStream is a Google Research neural audio codec from July 2021 that uses a convolutional encoder, residual vector quantization and a convolutional decoder.
  2. Its encoder turns 24 kHz audio into 75 embeddings per second, and each embedding is sent as a short list of codebook indices.
  3. Quantizer dropout lets one SoundStream model run anywhere from 3 to 18 kbps by using more or fewer quantizer layers.
  4. In listening tests SoundStream at 3 kbps outperformed Opus at 12 kbps and approached EVS at 9.6 kbps.
  5. Google's Lyra V2 codec and AudioLM generator both build on SoundStream, the first for calls and the second for audio tokens.

深入探討

A codec, short for coder-decoder, is software that shrinks audio for sending or storage and rebuilds it on the other side. Traditional speech codecs such as Opus and EVS are designed by hand using signal processing and models of human hearing. SoundStream, described by Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund and Marco Tagliasacchi of Google Research in a July 2021 paper, replaces those hand-built parts with neural networks trained end to end. It handles speech, music and general audio with one model and can stream, meaning it encodes and decodes as audio arrives rather than waiting for a whole file.

The pipeline has three parts. An encoder, a stack of one-dimensional convolutional layers, reads the raw waveform sampled at 24 kHz and downsamples it with strides of 2, 4, 5 and 8, producing one embedding vector for every 320 samples. That is 75 vectors per second, one every 13.3 milliseconds. A quantizer then replaces each vector with a short list of codebook indices, which is what actually gets transmitted. A decoder, built like the encoder in reverse with transposed convolutions, turns the indices back into a waveform. All three are trained together so that the output sounds like the input.

The quantizer is the key idea. Vector quantization maps a vector to the nearest entry in a learned codebook and sends only that entry's number. The paper shows why plain vector quantization fails at useful bitrates: at 6 kbps and 75 frames per second each frame gets 80 bits, which would require a codebook of 2 to the power 80 entries. SoundStream instead uses residual vector quantization (RVQ). The first quantizer picks the closest codebook entry, the difference between the vector and that entry, called the residual, goes to a second quantizer, its residual goes to a third, and so on. With 8 quantizers of 1,024 entries each, every quantizer spends 10 bits and the total is 80 bits per frame, using only 8,192 stored vectors in total.

Training combines two kinds of loss. A reconstruction loss compares spectrograms of the input and output at several time-frequency resolutions, and a feature loss compares how the discriminators respond to real and decoded audio. Adversarial losses come from discriminators, networks trained to tell real audio from decoded audio, which push the decoder toward natural-sounding output rather than a blurry average. A technique called quantizer dropout makes the bitrate adjustable: during training, the model randomly uses only the first n quantizers, so a single trained model can later run anywhere from 3 to 18 kbps, and the authors report negligible quality loss compared with models trained for one fixed bitrate.

The headline results come from crowd-sourced listening tests inspired by MUSHRA, a standard method in which listeners rate several versions of a clip against a hidden reference. On 200 clips of clean speech, noisy speech, reverberant speech and music, each rated 20 times, SoundStream at 3 kbps outperformed Opus at 12 kbps and approached EVS at 9.6 kbps, which Google's blog post describes as using 3.2 to 4 times fewer bits. The model also proved fast: on a single CPU thread of a Pixel 4 phone, the default configuration ran more than 2.3 times faster than real time, and a smaller version more than 7.1 times. Training used LibriTTS for clean speech, LibriTTS mixed with Freesound noise for noisy speech, and MagnaTagATune for music.

Compared with traditional codecs, SoundStream's strength is quality at very low bitrates and its ability to treat any kind of sound with one model. Its costs are the usual neural ones: it must be trained on large audio collections, and both ends must run the network rather than a compact hand-written algorithm. Compared with Google's first Lyra codec, an autoregressive generative model at 3 kbps, SoundStream is not autoregressive and was designed for low latency. Later codecs built on the same recipe include Meta's EnCodec, released in October 2022 for 24 kHz mono and 48 kHz stereo audio, and the Improved RVQGAN codec, which reports roughly 90 times compression of 44.1 kHz audio into 8 kbps.

The codec's limits are documented in its own paper and its successors. Quality falls as fewer quantizers are used, and latency is a trade-off: doubling the frame length to 26.6 ms requires 160 bits per frame for the same 6 kbps budget. Bitstreams are tied to a specific trained model, so both ends must run the same weights, unlike standardized codecs with fixed formats. The authors also note in the paper that PESQ and POLQA, the usual ITU-T quality metrics, are inconvenient to use because of licensing restrictions, so they relied on the open ViSQOL metric and human tests for evaluation.

SoundStream's second life is as an audio tokenizer. Because RVQ turns audio into a grid of discrete integers, those integers can be modeled like words. Google's AudioLM, announced in October 2022, uses SoundStream codes as acoustic tokens, generating them with language models and decoding them with the SoundStream decoder. In practice, developers who want SoundStream for calls can use Lyra V2, released under the Apache license in September 2022 for Android and Linux with experimental Mac and Windows support.

04工作範例

Working out SoundStream's bitrate from its settings

  1. Start with the sample rate, 24,000 samples per second, and the encoder's total stride, 2 x 4 x 5 x 8 = 320.

  2. Divide to get frames per second: 24,000 / 320 = 75 embeddings each second, one every 13.3 ms.

  3. Pick a codebook size of 1,024 entries, which needs log2(1,024) = 10 bits per quantizer index.

  4. Choose 4 quantizer layers: 75 x 4 x 10 = 3,000 bits per second, or 3 kbps.

  5. Choose 8 layers instead: 75 x 8 x 10 = 6,000 bits per second, or 6 kbps.

  6. Choose 24 layers: 75 x 24 x 10 = 18,000 bits per second, the top of SoundStream's 3 to 18 kbps range.

它顯示了什麼

The learner can now predict the bitrate of any RVQ codec from three numbers: frame rate, number of quantizers and bits per codebook. A check: Lyra V2's 3.2, 6 and 9.2 kbps modes are chosen the same way, by changing the number of quantizers used.

現實世界的實施

Google's Lyra V2 speech codec, released as open source in September 2022, is built on SoundStream and offers 3.2, 6 and 9.2 kbps modes, with 20 ms latency and encoding plus decoding of a 20 ms frame in 0.57 ms on a Pixel 6 Pro.

Google's AudioLM, announced in October 2022, uses SoundStream codes as its acoustic tokens, generating speech and piano continuations as token sequences and then decoding them back to waveforms with the SoundStream decoder.

Low-bandwidth voice calls are the original target: the SoundStream paper reports that at 3 kbps it outperformed Opus at 12 kbps and approached EVS at 9.6 kbps in crowd-sourced listening tests on 24 kHz audio.

One SoundStream variant performs compression and background noise removal in the same model, switched on or off by a conditioning signal, so a call app could clean up speech without adding latency.

Meta's EnCodec (2022) and the Improved RVQGAN codec (NeurIPS 2023) follow the same encoder, residual quantizer and adversarial decoder recipe, the latter compressing 44.1 kHz audio about 90 times to 8 kbps.

The Future of SoundStream Neural Audio Codec

SoundStream's design has become a template. Meta's EnCodec (October 2022) and the Improved RVQGAN codec (NeurIPS 2023 spotlight) both pair a convolutional encoder and decoder with residual vector quantization and adversarial training, extending the idea to 48 kHz stereo and to roughly 90 times compression of 44.1 kHz audio. Lyra V2 (September 2022) showed that the approach can ship as an open-source, real-time product with 20 ms latency. The larger shift is that neural codecs now serve generative AI as much as communication. AudioLM (2022) treated SoundStream codes as a vocabulary for language models, and later speech and music generators adopted similar codec tokens. That raises open questions about how many tokens per second a generator should model, how to separate content from speaker identity, and how to keep codes useful for both compression and generation. Open problems for real-world use include robustness to unusual audio, fairness across languages and voices, interoperability, since each trained model defines its own bitstream, and evaluation, since the SoundStream authors relied on human tests and the open ViSQOL metric rather than licensed ITU-T scores. Detecting synthetic audio produced through codec-based generators is also an active area, as Google's AudioLM work noted with its own detection classifier.

付諸實踐

為什麼這很重要

  • 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
  • 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
  • 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。

留意

  • 如果未徵得同意,語音濫用和冒充風險就會增加。
  • 由於口音、方言或嘈雜的環境,準確性可能會下降。
  • 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。

後續步驟

  1. 獲得語音捕獲、克隆和重用的明確同意。
  2. 測試不同揚聲器和背景條件下的品質。
  3. 定義人員必須審查或批准輸出的時間。
  4. 標記合成音訊並保留來源記錄以供問責。

資料來源與延伸閱讀

  1. arXivSoundStream:端對端神經音訊編解碼器
  2. Google Research BlogSoundStream:端對端神經音訊編解碼器
  3. Google Open Source BlogLyra V2: a better, faster, and more versatile speech codec
  4. Google Research BlogAudioLM:音訊產生的語言建模方法
  5. arXiv高保真神經音訊壓縮
  6. arXiv (NeurIPS 2023)High-Fidelity Audio Compression with Improved RVQGAN

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the SoundStream Neural Audio Codec quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

What is SoundStream Neural Audio Codec?

SoundStream is a neural audio codec from Google Research, published in July 2021, that compresses speech, music and general audio with a learned encoder, a residual vector quantizer and a learned decoder. In listening tests it matched or beat traditional codecs at a fraction of their bitrate, and its discrete codes became a standard way to turn audio into tokens for generative models.

SoundStream 架構由哪三個元件組成?

SoundStream 由卷積編碼器、離散化嵌入的殘差向量量化器和卷積解碼器構建,所有這些都經過端到端訓練。

什麼技術可以讓單一 SoundStream 模型在無需重新訓練的情況下提供多種不同的位元率?

在訓練期間,SoundStream 會隨機丟棄量化器層,以便在推理時可以使用更多或更少的 RVQ 等級來達到一個模型的不同位元率。

在 Google 的聆聽測試中,據報告,3 kbps 的 SoundStream 效能優於 12 kbps 的哪種編解碼器?

在主觀品質評估中,僅 3 kbps 的 SoundStream 與以 12 kbps 運行的廣泛使用的 Opus 編解碼器相匹配或擊敗。

SoundStream 使用什麼樣的附加訓練訊號來讓輸出聲音自然?

除了重建損失之外,SoundStream 還使用對抗性 (GAN) 判別器,因此重建聽起來感覺上很真實,而不僅僅是數字上接近。

SoundStream 的比特率與其量化器有何關係?

位元率隨著活動 RVQ 層的數量而變化(乘以幀速率乘以每個碼本的位數),因此添加量化器可以提高位元率和品質。