音訊人工智慧指南

使用 RNNoise 進行語音去噪

RNNoise 是一個開源的雜訊抑制庫,由 Xiph.Org 的 Jean-Marc Valin 與 Mozilla 於 2017 年發布,可即時去除語音中的背景雜訊。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 使用 RNNoise 示範程式替錄音降噪
  5. 現實世界的實施
  6. RNNoise 語音降噪的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

它將傳統的訊號處理與擁有約 88,000 個權重的小型循環神經網路相結合,因此可在一般的 CPU 核心上執行,並成為串流與語音通話軟體中的常見建構區塊。

重點摘要

  1. RNNoise 是來自 Xiph.Org 與 Mozilla、採 BSD 授權的 C 語言庫,於 2017 年發布,可即時抑制 48 kHz 語音中的背景雜訊。
  2. 它使用小型閘控循環神經網路來預測 22 個頻帶增益,而不是估算每個頻率區間,這使計算量保持在很低的水準。
  3. 2018 年的論文指出其擁有 87,503 個權重,僅佔一個 x86 核心約 1.3% 的資源,且不需要 GPU。
  4. 音高調校的梳狀濾波器可移除粗略頻帶增益無法觸及的語音泛音之間的雜訊。
  5. OBS Studio 在 2020 年 9 月發布的 26.0 版本中將 RNNoise 新增為雜訊抑制方法。

深入探討

RNNoise is a speech enhancement library: software that takes audio containing someone talking over background noise, such as a fan, traffic or keyboard clicks, and returns a cleaner version in which the voice is kept and the noise is reduced. It was written by Jean-Marc Valin and published in September 2017 as a Xiph.Org and Mozilla project with a browser demo and a technical paper. The paper, A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement, appeared at the 2018 IEEE Workshop on Multimedia Signal Processing. The code is written in C and distributed under a BSD license.

The design goal was practical: run in real time on modest hardware at full-band 48 kHz audio, the sample rate used in high-quality voice calls, without a GPU. Many deep learning denoisers of the time estimated every frequency bin of a spectrogram, or even every audio sample, which takes large networks. RNNoise instead keeps the parts of a traditional noise suppressor that signal processing already does well, such as framing, spectral analysis and pitch tracking, and asks a neural network to do only the hardest part, which is estimating how much each part of the spectrum should be turned down.

The processing happens in short, overlapping frames. Audio is cut into 20 millisecond windows that advance every 10 milliseconds, and each window is converted to the frequency domain with a Fourier transform. Rather than handle hundreds of individual frequency bins, RNNoise groups them into 22 bands spaced on the Bark scale, a scale that roughly matches how human hearing resolves frequency, using the band layout of the Opus codec. For each frame it computes 42 input features: 22 Bark-frequency cepstral coefficients that summarize the spectral shape, the first and second time derivatives of the first six, six coefficients describing how strongly each band follows the voice's pitch, the pitch period itself and a measure of how stationary the spectrum is.

Those features go into a recurrent neural network, a network that keeps an internal state from frame to frame so it can track how sounds evolve over time. The network uses gated recurrent units (GRUs), a type of recurrent layer that learns what to remember and what to forget, combined with fully connected layers. The paper reports four hidden layers with 215 units in total, the largest having 96, and 87,503 weights. The network produces 22 gains between 0 and 1, one per band, plus a voice activity probability. Each gain multiplies the energy in its band: a value near 1 lets the band through, near 0 suppresses it. Because 22 band gains are too coarse to remove noise sitting between the harmonics of a voice, a final pitch filter, a comb filter tuned to the detected pitch, attenuates energy between harmonics. The result is converted back to a waveform by overlap-add.

Training uses synthetic mixtures. Clean speech and noise are rarely recorded together with a clean reference, so they are recorded separately and mixed at random levels with random filtering. The paper reports about 6 hours of speech and 4 hours of noise expanded this way into about 140 hours of noisy training audio. The target for each band is the square root of the ratio between clean speech energy and noisy energy, and the loss compares gains after a further square root, which penalizes mistakes on quiet bands more. Because the system does not normalize levels, the training data must include speech at every realistic volume.

The efficiency numbers explain its adoption. The paper reports that the network needs about 175,000 floating point operations per frame, or 17.5 million per second at 100 frames per second, and about 40 million per second for the whole algorithm including transforms and pitch search. That used about 1.3 percent of one core of an Intel Haswell laptop processor and about 14 percent of a Raspberry Pi 3 core. Weights constrained to plus or minus 0.5 during training can be stored as 8-bit values, giving a model of about 85 kB according to the demo page. In a PESQ evaluation, an objective speech quality score, the paper reports clearly better quality than the conventional suppressor in SpeexDSP, particularly for non-stationary noise such as babble, car and street noise.

Compared with the alternatives, RNNoise sits between old and new. Classical suppressors such as the Speex preprocessor estimate the noise spectrum statistically and subtract it; they are cheap but struggle with noise that changes quickly. Large neural denoisers trained for the Microsoft-run Deep Noise Suppression Challenge at INTERSPEECH 2020 can remove more noise but cost more compute; the challenge organizers ranked entries with crowdsourced listening tests under the ITU-T P.808 standard because objective metrics often disagreed with human judgments. Valin's own follow-up with co-authors, PercepNet, presented at INTERSPEECH 2020, kept the band-and-pitch design but reported higher quality while still using less than 5 percent of a CPU core.

The limitations follow from the design. A band gain cannot separate a voice from noise that occupies the same band at the same moment, so strong noise leaves a muffled or watery voice, a distortion often called musical noise. RNNoise is trained to separate speech from background sounds, not one talker from another, so a competing voice in the background is treated as speech. Its quality is bounded by the training mixtures, and a 2017-era model can misjudge noise types it never heard. The 10 millisecond look-ahead keeps latency low for calls but limits how much future context the network can use.

Using RNNoise in practice requires little. The repository builds with a standard configure and make process, downloads the model files during setup, and includes a demo that reads headerless 16-bit mono audio at 48 kHz. Developers who want a custom model can generate features from their own speech and noise files with the provided tools, train with the included Python script and compile the quantized weights back into the library. The README recommends at least 10,000 training sequences and preferably 200,000 or more.

04工作範例

使用 RNNoise 示範程式替錄音降噪

  1. 錄製約 30 秒的語音,環境靠近運轉的風扇,或將乾淨語音與噪音片段混合。

  2. 將檔案轉為原始無標頭的 16 位元單聲道 PCM,採樣率 48 kHz,例如使用指令列音訊轉換工具。

  3. 從官方倉庫使用 autogen.sh 取得原始碼,執行 configure 與 make 編譯,同時會下載模型檔。

  4. 執行 examples/rnnoise_demo,將嘈雜檔案作為輸入,並指定新檔名作為輸出。

  5. 將輸出檔轉回 WAV,並交替聆聽原始與去噪後的版本。

  6. 在頻譜圖檢視器中同時開啟兩個檔案,比較語音諧波與停頓期間的能量差異。

它顯示了什麼

學習者最終會得到一個去噪後的檔案,其中穩定的風扇噪音被大幅降低,特別是在停頓處。頻譜圖應顯示語音諧波之間有更暗的間隙,而與語音重疊的極大噪音可能會留下一些悶音,這說明了頻帶增益的限制。

現實世界的實施

於 2020 年 9 月發布的 OBS Studio 26.0 在其「噪音抑制」濾鏡中加入了 RNNoise 作為一種方法,為實況主提供了除了舊版基於 Speex 的抑制器之外的另一種神經網路選項。

GamingOnLinux 報導稱,Linux 工具 NoiseTorch 和 Cadmus 使用相同的 RNNoise 技術,在系統層級清理麥克風輸入,以用於語音聊天和通話。

官方儲存庫中的 rnnoise_demo 程式可從命令列對原始的 16-bit、48 kHz 單聲道錄音進行去噪,這使得在 Podcast 或會議音訊上進行測試變得非常容易。

Valin 的「噪音捐贈」(Noise Donation) 專案邀請大眾錄製一分鐘自己環境的噪音,收集到的錄音被發布為一個 6.4 GB 的可下載壓縮檔,用於訓練。

於 2024 年 4 月發布的 RNNoise 0.2,僅使用公開可用的資料集重新訓練了模型,並增加了 AVX2 和 SSE4.1 優化以及執行時 CPU 偵測。

RNNoise 語音降噪的未來

RNNoise's 2024 release, version 0.2, changed training and engineering rather than the core idea: Phoronix reported in April 2024 that the distributed models were retrained using only publicly available datasets and that the code gained SSE4.1 and AVX2 optimizations with run-time CPU detection. The repository now also offers a sparser little model and a way to load weights from a file at run time, which makes swapping in custom models easier. The broader research direction, visible in PercepNet (INTERSPEECH 2020) and the Deep Noise Suppression Challenge series that began in 2020, is toward higher perceived quality at similar low compute, judged by human listening tests rather than objective scores alone, since the challenge organizers found the two often disagree. Open problems include suppressing competing talkers and reverberation without damaging the target voice, robustness to noise types absent from training data, and evaluation that reflects real calls rather than synthetic mixtures. Whether small hybrid designs like RNNoise or larger fully neural models running on dedicated hardware become the default for voice calls remains an open question.

付諸實踐

為什麼這很重要

  • 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
  • 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
  • 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。

留意

  • 如果未徵得同意,語音濫用和冒充風險就會增加。
  • 由於口音、方言或嘈雜的環境,準確性可能會下降。
  • 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。

後續步驟

  1. 獲得語音捕獲、克隆和重用的明確同意。
  2. 測試不同揚聲器和背景條件下的品質。
  3. 定義人員必須審查或批准輸出的時間。
  4. 標記合成音訊並保留來源記錄以供問責。

資料來源與延伸閱讀

  1. arXiv / IEEE MMSP 2018一種用於即時全頻語音增強的混合 DSP/深度學習方法
  2. Jean-Marc Valin (Xiph.Org / Mozilla)RNNoise:學習噪音抑制
  3. Xiph.Org Foundation (GitHub)xiph/rnnoise:用於音訊降噪的遞迴神經網路
  4. PhoronixRNNoise 0.2 發布,針對神經網路噪音抑制進行了 AVX2 優化
  5. GamingOnLinuxOBS Studio 在 26.0 版本中透過 RNNoise 加入了更好的噪音抑制功能
  6. arXiv / INTERSPEECH 2020一種用於低複雜度、即時全頻語音增強的感知驅動方法
  7. arXiv / INTERSPEECH 2020INTERSPEECH 2020 深度噪音抑制挑戰賽:資料集、主觀測試框架與挑戰賽結果

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加「使用 RNNoise 進行語音降噪」測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是使用 RNNoise 進行語音降噪?

RNNoise 是一個開源噪音抑制函式庫,由 Xiph.Org 和 Mozilla 的 Jean-Marc Valin 於 2017 年發布,能即時消除語音中的背景噪音。它將傳統訊號處理與一個約 88,000 個權重的小型遞迴神經網路相結合,因此可以在普通的 CPU 核心上執行,並成為串流媒體和語音通話軟體中常見的建構模組。

RNNoise 對每個短音頻幀主要預測什麼?

RNNoise 估計每個頻帶的增益,這些增益會衰減噪音主導的頻帶,同時保留語音主導的頻帶,而不是在每個頻譜段上工作。

RNNoise 的核心是什麼類型的神經網路?

RNNoise 使用由門控循環單元構建的緊湊循環神經網絡,在保持微小的同時賦予其時間記憶。

為什麼 RNNoise 使用感知頻帶而不是原始頻譜箱?

僅預測約 22 個頻帶增益可以使網路保持小型、快速,且不易受到困擾 per-bin 方法的音樂噪音偽影的影響。

RNNoise 模型大約有多大?

RNNoise 具有大約 85,000 個權重,足夠小,可以在單一 CPU 核心上比即時運行得更快。

RNNoise 中音調濾波器的作用是什麼?

梳狀/音調濾波器利用有聲語音的諧波結構來抑制諧波之間的噪聲,從而補充頻帶增益。