双路径 RNN 分离

Speech separation is the task of taking one recording in which several people talk at once and producing one clean track per speaker, often called the cocktail party problem. Dual-path recurrent neural network, or DPRNN, is a way of organizing the network that does this. It was introduced by Yi Luo of Columbia University with Zhuo Chen and Takuya Yoshioka of Microsoft in the paper "Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation", posted in October 2019 and presented at ICASSP 2020. Its contribution is not a new kind of neuron but a new arrangement that lets ordinary recurrent networks handle extremely long inputs.

The problem it solves comes from time-domain separation. Earlier systems worked on spectrograms, but Conv-TasNet, published by Yi Luo and Nima Mesgarani in 2018, showed that learning directly from the waveform worked better. Its design has three parts: a learned encoder that turns short windows of audio into feature vectors, a separator that estimates a mask (a set of weights saying how much of each feature belongs to each speaker), and a decoder that turns masked features back into audio. Shorter encoder windows give better separation, but a 4-second clip at 8 kHz holds 32,000 samples, and with a 2-sample window the separator must model tens of thousands of steps. Recurrent networks, which read a sequence one step at a time while carrying a memory, are hard to train over sequences that long. Convolutional separators only see a fixed stretch of context.

DPRNN works in three stages. First, segmentation: the long sequence of feature vectors is cut into chunks of length K that overlap by half, and the chunks are stacked into a three-dimensional block, like pages of a book. Second, block processing: an intra-chunk recurrent network reads along each chunk to model local detail such as the shape of a syllable, and then an inter-chunk recurrent network reads across chunks at the same position to connect distant parts of the utterance. These two passes form one DPRNN block, and the paper stacks six of them, each using bidirectional LSTM layers with 128 hidden units in each direction. Third, overlap-add: the chunks are laid back out in order and overlapping parts are summed, restoring the original length. If K is chosen near the square root of twice the sequence length, both recurrent networks see sequences of only a few hundred steps instead of tens of thousands.

The results were striking for the model's size. On WSJ0-2mix, a standard benchmark of artificial two-speaker mixtures from Wall Street Journal read speech with 30 hours of 8 kHz training audio, swapping Conv-TasNet's convolutional separator for DPRNN raised scale-invariant signal-to-noise ratio improvement (SI-SNRi, how many decibels cleaner the output is than the mixture) from 15.2 to 16.0 dB while nearly halving the model from 5.1 to 2.6 million parameters. Shrinking the encoder window from 16 samples to 2, which DPRNN made trainable, raised SI-SNRi to 18.8 dB. The previous best system, FurcaNeXt, used 51.4 million parameters, 20 times more. For comparison, deep clustering, the 2015 method by John Hershey and colleagues that introduced this two-speaker evaluation, improved signal quality by about 6 dB.

DPRNN changed the field mostly through its layout. In 2020 the Dual-Path Transformer Network (DPTNet) kept the chunk structure but used Transformer layers combined with a recurrent network, reporting 20.6 dB signal-to-distortion ratio on WSJ0-2mix. SepFormer, from Cem Subakan and colleagues at ICASSP 2021, explicitly adopted DPRNN's dual-scale approach with an IntraTransformer and an InterTransformer and no recurrence at all, reaching 22.3 dB SI-SNRi with data augmentation. The authors noted that DPRNN's recurrent connections, especially in the global pass, limit parallel computation, which is the main reason later work replaced them.

Compared with alternatives, DPRNN is small and accurate but sequential. Convolutional separators like Conv-TasNet are fast and parallel but see limited context. Transformer dual-path models such as SepFormer are more accurate and parallelize better on GPUs but are much larger, 26 million parameters against 2.6 million. Multi-microphone methods such as beamforming use spatial information DPRNN does not have, and DPRNN works with a single channel. Spectrogram-based methods are easier to interpret but were outperformed on clean benchmarks by time-domain models from Conv-TasNet onward.

The main limitation is the gap between benchmark and reality. WSJ0-2mix is clean read speech mixed artificially, and real rooms add noise and reverberation. On the paper's own noisy, reverberant LibriSpeech mixtures, DPRNN achieved 8.4 dB SI-SNRi, far below its 18.8 dB on clean data, and a speech recognizer still made 25.9% word errors on its output against 9.1% on reverberant speech with no overlap. Standard DPRNN models are trained for a fixed number of speakers, so a two-speaker model cannot handle three talkers without changes. Very short encoder windows make sequences long and computation heavy even with chunking, and recurrent layers process steps in order, which slows inference on GPUs.

In practice DPRNN is easy to reproduce. The paper trained on 4-second segments for 100 epochs with the Adam optimizer, a starting learning rate of 0.001 decayed by 0.98 every two epochs, and utterance-level permutation invariant training, which scores every possible matching of outputs to speakers and trains on the best one. Open implementations exist in the Asteroid toolkit, which also covers the noisy WHAM and WHAMR datasets and LibriMix. Building a real product still requires training data that resembles the target setting, since models trained only on WSJ0-2mix degrade on real recordings.