音訊人工智慧指南

雙路徑 RNN 分離

雙路徑循環神經網路(DPRNN)是一種用於分離單麥克風錄音中重疊人聲的神經網路設計。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 為 DPRNN 分離器設定區塊大小
  5. 現實世界的實施
  6. 雙路徑 RNN 分離的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

它將非常長的序列切割成短區塊,並在讀取每個區塊內部的循環網路與讀取跨區塊的循環網路之間交替進行。該方法於 2019 年提出,在 WSJ0-2mix 基準測試中創下紀錄,其模型比之前的最佳模型小 20 倍,且其雙路徑架構成為後續分離模型的範本。

重點摘要

  1. DPRNN 將長特徵序列分割成重疊的區塊,並在區塊內部的循環傳遞與區塊之間的循環傳遞之間交替進行。
  2. 選擇接近序列長度兩倍平方根的區塊長度,可使每次循環傳遞僅需處理數百個步驟。
  3. DPRNN-TasNet 在 WSJ0-2mix 上達到了 18.8 dB 的 SI-SNRi,參數量為 260 萬,比之前的最佳模型小 20 倍。
  4. SepFormer 保留了 DPRNN 的雙路徑架構,但將循環結構替換為 Transformers,在 WSJ0-2mix 上以 2600 萬參數達到了 22.3 dB。
  5. 在充滿噪音和混響的混合訊號中,DPRNN 得分為 8.4 dB,顯示出乾淨基準測試與真實錄音之間存在巨大差距。

深入探討

Speech separation is the task of taking one recording in which several people talk at once and producing one clean track per speaker, often called the cocktail party problem. Dual-path recurrent neural network, or DPRNN, is a way of organizing the network that does this. It was introduced by Yi Luo of Columbia University with Zhuo Chen and Takuya Yoshioka of Microsoft in the paper "Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation", posted in October 2019 and presented at ICASSP 2020. Its contribution is not a new kind of neuron but a new arrangement that lets ordinary recurrent networks handle extremely long inputs.

The problem it solves comes from time-domain separation. Earlier systems worked on spectrograms, but Conv-TasNet, published by Yi Luo and Nima Mesgarani in 2018, showed that learning directly from the waveform worked better. Its design has three parts: a learned encoder that turns short windows of audio into feature vectors, a separator that estimates a mask (a set of weights saying how much of each feature belongs to each speaker), and a decoder that turns masked features back into audio. Shorter encoder windows give better separation, but a 4-second clip at 8 kHz holds 32,000 samples, and with a 2-sample window the separator must model tens of thousands of steps. Recurrent networks, which read a sequence one step at a time while carrying a memory, are hard to train over sequences that long. Convolutional separators only see a fixed stretch of context.

DPRNN works in three stages. First, segmentation: the long sequence of feature vectors is cut into chunks of length K that overlap by half, and the chunks are stacked into a three-dimensional block, like pages of a book. Second, block processing: an intra-chunk recurrent network reads along each chunk to model local detail such as the shape of a syllable, and then an inter-chunk recurrent network reads across chunks at the same position to connect distant parts of the utterance. These two passes form one DPRNN block, and the paper stacks six of them, each using bidirectional LSTM layers with 128 hidden units in each direction. Third, overlap-add: the chunks are laid back out in order and overlapping parts are summed, restoring the original length. If K is chosen near the square root of twice the sequence length, both recurrent networks see sequences of only a few hundred steps instead of tens of thousands.

The results were striking for the model's size. On WSJ0-2mix, a standard benchmark of artificial two-speaker mixtures from Wall Street Journal read speech with 30 hours of 8 kHz training audio, swapping Conv-TasNet's convolutional separator for DPRNN raised scale-invariant signal-to-noise ratio improvement (SI-SNRi, how many decibels cleaner the output is than the mixture) from 15.2 to 16.0 dB while nearly halving the model from 5.1 to 2.6 million parameters. Shrinking the encoder window from 16 samples to 2, which DPRNN made trainable, raised SI-SNRi to 18.8 dB. The previous best system, FurcaNeXt, used 51.4 million parameters, 20 times more. For comparison, deep clustering, the 2015 method by John Hershey and colleagues that introduced this two-speaker evaluation, improved signal quality by about 6 dB.

DPRNN changed the field mostly through its layout. In 2020 the Dual-Path Transformer Network (DPTNet) kept the chunk structure but used Transformer layers combined with a recurrent network, reporting 20.6 dB signal-to-distortion ratio on WSJ0-2mix. SepFormer, from Cem Subakan and colleagues at ICASSP 2021, explicitly adopted DPRNN's dual-scale approach with an IntraTransformer and an InterTransformer and no recurrence at all, reaching 22.3 dB SI-SNRi with data augmentation. The authors noted that DPRNN's recurrent connections, especially in the global pass, limit parallel computation, which is the main reason later work replaced them.

Compared with alternatives, DPRNN is small and accurate but sequential. Convolutional separators like Conv-TasNet are fast and parallel but see limited context. Transformer dual-path models such as SepFormer are more accurate and parallelize better on GPUs but are much larger, 26 million parameters against 2.6 million. Multi-microphone methods such as beamforming use spatial information DPRNN does not have, and DPRNN works with a single channel. Spectrogram-based methods are easier to interpret but were outperformed on clean benchmarks by time-domain models from Conv-TasNet onward.

The main limitation is the gap between benchmark and reality. WSJ0-2mix is clean read speech mixed artificially, and real rooms add noise and reverberation. On the paper's own noisy, reverberant LibriSpeech mixtures, DPRNN achieved 8.4 dB SI-SNRi, far below its 18.8 dB on clean data, and a speech recognizer still made 25.9% word errors on its output against 9.1% on reverberant speech with no overlap. Standard DPRNN models are trained for a fixed number of speakers, so a two-speaker model cannot handle three talkers without changes. Very short encoder windows make sequences long and computation heavy even with chunking, and recurrent layers process steps in order, which slows inference on GPUs.

In practice DPRNN is easy to reproduce. The paper trained on 4-second segments for 100 epochs with the Adam optimizer, a starting learning rate of 0.001 decayed by 0.98 every two epochs, and utterance-level permutation invariant training, which scores every possible matching of outputs to speakers and trains on the best one. Open implementations exist in the Asteroid toolkit, which also covers the noisy WHAM and WHAMR datasets and LibriMix. Building a real product still requires training data that resembles the target setting, since models trained only on WSJ0-2mix degrade on real recordings.

04工作範例

為 DPRNN 分離器設定區塊大小

  1. 從 8 kHz 採樣的 4 秒雙人混合語音開始,即 32,000 個採樣點。

  2. 使用 2 個採樣點的編碼器視窗,步長(hop)為 1 個採樣點,產生約 32,000 個特徵幀(L)。

  3. 計算區塊大小:2 的平方根乘以 32,000 約為 253,因此選擇 K = 250,步長為 125。

  4. 計算區塊數:約 32,000 / 125,大約為 256 個區塊(S),因此當編碼器具有 64 個濾波器時,張量大小為 64 個特徵乘以 250 再乘以 256。

  5. 追蹤一個區塊:區塊內 LSTM 在 250 個步驟上運行 256 次,然後區塊間 LSTM 在 256 個步驟上運行 250 次。

  6. 相較於普通的 LSTM 需要對 32,000 個步驟進行一次遍歷,請注意,任何 LSTM 所看到的的最長路徑現在大約是 256 個步驟。

它顯示了什麼

一個符合論文最佳 WSJ0-2mix 設定(視窗 2、區塊 250)的區塊組態。若要檢查它,請執行諸如 Asteroid 的 DPRNN 實作,並確認分離器的內部張量具有預測的形狀,然後在保留的混合音訊上測量 SI-SNRi。

現實世界的實施

在 2019 年的論文中,DPRNN-TasNet 使用 260 萬個參數,以 18.8 dB 的尺度不變訊號雜訊比改善(SI-SNRi),分離了 WSJ0-2mix 中的雙語者混合音訊。

在由 LibriSpeech 構建的嘈雜、具迴響的雙語者混合音訊上,同一篇論文將語音辨識字詞錯誤率從 Conv-TasNet 的卷積分離器的 28.7% 降至 25.9%。

用於 PyTorch 的開源 Asteroid 工具包包含一個 DualPathRNN 實作,以及針對嘈雜 WHAM 資料集的訓練配方,同時也包含了 Conv-TasNet、DPTNet 和其他分離模型。

SepFormer (ICASSP 2021) 保留了 DPRNN 基於區塊的區塊內與區塊間配置,但將循環神經網路替換為 Transformer,在 WSJ0-2mix 上達到 22.3 dB;可從 Hugging Face 下載 SpeechBrain 檢查點。

具備跨區塊單向循環網路的因果版本,可讓 DPRNN 以由區塊大小決定的延遲執行,這適合如即時字幕等串流應用。

雙路徑 RNN 分離的未來

The dual-path idea outlived its recurrent networks. After DPRNN in 2019, DPTNet in 2020 and SepFormer in 2021 kept the chunk-wise local and global split while moving to attention, and SepFormer's authors cited speed and memory advantages over recurrent models with comparable quality. Open problems in single-channel separation are less about architecture than about realism: models score above 20 dB on clean synthetic mixtures but far lower in noisy, reverberant rooms, as DPRNN's own 8.4 dB result on reverberant LibriSpeech mixtures showed. Research directions include training on more realistic data such as the WHAM and WHAMR noisy and reverberant extensions available in Asteroid, handling an unknown or changing number of speakers in long meetings, and reducing compute for on-device use, where DPRNN's 2.6-million-parameter size remains attractive. Joint training with speech recognition, so separation is judged by transcription accuracy rather than signal metrics, is another active line. Which of these will matter most for products such as meeting transcription and hearing assistance is not settled.

付諸實踐

為什麼這很重要

  • 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
  • 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
  • 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。

留意

  • 如果未徵得同意,語音濫用和冒充風險就會增加。
  • 由於口音、方言或嘈雜的環境,準確性可能會下降。
  • 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。

後續步驟

  1. 獲得語音捕獲、克隆和重用的明確同意。
  2. 測試不同揚聲器和背景條件下的品質。
  3. 定義人員必須審查或批准輸出的時間。
  4. 標記合成音訊並保留來源記錄以供問責。

資料來源與延伸閱讀

  1. arXiv (ICASSP 2020)雙路徑 RNN:針對時域單聲道語音分離的高效長序列建模
  2. arXiv (IEEE/ACM TASLP)Conv-TasNet:超越理想時頻振幅遮罩的語音分離
  3. arXiv深度分群:用於分割與分離的區別性嵌入
  4. arXiv (Interspeech 2020)雙路徑 Transformer 網路:用於端對端單耳語音分離的直接上下文感知建模
  5. arXiv (ICASSP 2021)語音分離中的 Attention is All You Need
  6. GitHub (Asteroid team)Asteroid:基於 PyTorch 的研究人員音訊來源分離工具包
  7. Hugging Face (SpeechBrain)speechbrain/sepformer-wsj02mix

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加雙路徑 RNN 分離測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是雙路徑 RNN 分離?

雙路徑 RNN (DPRNN) 是一種神經網路設計,用於分離單麥克風錄音中的重疊人聲。它將非常長的序列切割成短區塊,並在讀取每個區塊內部的循環網路與讀取區塊之間的循環網路之間交替。它於 2019 年推出,以一個比先前最佳模型小 20 倍的模型在 WSJ0-2mix 基準測試中創下紀錄,其雙路徑配置成為後續分離模型的範本。

Dual-Path RNN主要解決什麼問題?

DPRNN 將非常長的時域序列重新組織成區塊,因此 RNN 可以處理本地和全域上下文,而不會因長度而阻塞。

雙路徑 RNN 中的兩條「路徑」是什麼?

每個區塊內有一個 RNN 處理局部模式;跨區塊的其他過程用於遠端結構。

雙路徑區塊之前的長特徵序列是如何準備的?

DPRNN 將長序列折疊成重疊區塊的矩陣,因此每個 RNN 只處理一個短視窗。

DPRNN 作為分隔符號替代品被放入哪個現有框架中?

DPRNN 取代了 Conv-TasNet 管線內的 TCN 分離器,保留了學習到的編碼器和解碼器。

後來哪個模型保留了 DPRNN 的雙路徑結構,但用 Transformer 取代了 RNN?

SepFormer 重用了區塊內/區塊間雙路徑設計,但將循環單元替換為 Transformer 自註意力。