RNNoise를 사용한 음성 잡음 제거

RNNoise is a speech enhancement library: software that takes audio containing someone talking over background noise, such as a fan, traffic or keyboard clicks, and returns a cleaner version in which the voice is kept and the noise is reduced. It was written by Jean-Marc Valin and published in September 2017 as a Xiph.Org and Mozilla project with a browser demo and a technical paper. The paper, A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement, appeared at the 2018 IEEE Workshop on Multimedia Signal Processing. The code is written in C and distributed under a BSD license.

The design goal was practical: run in real time on modest hardware at full-band 48 kHz audio, the sample rate used in high-quality voice calls, without a GPU. Many deep learning denoisers of the time estimated every frequency bin of a spectrogram, or even every audio sample, which takes large networks. RNNoise instead keeps the parts of a traditional noise suppressor that signal processing already does well, such as framing, spectral analysis and pitch tracking, and asks a neural network to do only the hardest part, which is estimating how much each part of the spectrum should be turned down.

The processing happens in short, overlapping frames. Audio is cut into 20 millisecond windows that advance every 10 milliseconds, and each window is converted to the frequency domain with a Fourier transform. Rather than handle hundreds of individual frequency bins, RNNoise groups them into 22 bands spaced on the Bark scale, a scale that roughly matches how human hearing resolves frequency, using the band layout of the Opus codec. For each frame it computes 42 input features: 22 Bark-frequency cepstral coefficients that summarize the spectral shape, the first and second time derivatives of the first six, six coefficients describing how strongly each band follows the voice's pitch, the pitch period itself and a measure of how stationary the spectrum is.

Those features go into a recurrent neural network, a network that keeps an internal state from frame to frame so it can track how sounds evolve over time. The network uses gated recurrent units (GRUs), a type of recurrent layer that learns what to remember and what to forget, combined with fully connected layers. The paper reports four hidden layers with 215 units in total, the largest having 96, and 87,503 weights. The network produces 22 gains between 0 and 1, one per band, plus a voice activity probability. Each gain multiplies the energy in its band: a value near 1 lets the band through, near 0 suppresses it. Because 22 band gains are too coarse to remove noise sitting between the harmonics of a voice, a final pitch filter, a comb filter tuned to the detected pitch, attenuates energy between harmonics. The result is converted back to a waveform by overlap-add.

Training uses synthetic mixtures. Clean speech and noise are rarely recorded together with a clean reference, so they are recorded separately and mixed at random levels with random filtering. The paper reports about 6 hours of speech and 4 hours of noise expanded this way into about 140 hours of noisy training audio. The target for each band is the square root of the ratio between clean speech energy and noisy energy, and the loss compares gains after a further square root, which penalizes mistakes on quiet bands more. Because the system does not normalize levels, the training data must include speech at every realistic volume.

The efficiency numbers explain its adoption. The paper reports that the network needs about 175,000 floating point operations per frame, or 17.5 million per second at 100 frames per second, and about 40 million per second for the whole algorithm including transforms and pitch search. That used about 1.3 percent of one core of an Intel Haswell laptop processor and about 14 percent of a Raspberry Pi 3 core. Weights constrained to plus or minus 0.5 during training can be stored as 8-bit values, giving a model of about 85 kB according to the demo page. In a PESQ evaluation, an objective speech quality score, the paper reports clearly better quality than the conventional suppressor in SpeexDSP, particularly for non-stationary noise such as babble, car and street noise.

Compared with the alternatives, RNNoise sits between old and new. Classical suppressors such as the Speex preprocessor estimate the noise spectrum statistically and subtract it; they are cheap but struggle with noise that changes quickly. Large neural denoisers trained for the Microsoft-run Deep Noise Suppression Challenge at INTERSPEECH 2020 can remove more noise but cost more compute; the challenge organizers ranked entries with crowdsourced listening tests under the ITU-T P.808 standard because objective metrics often disagreed with human judgments. Valin's own follow-up with co-authors, PercepNet, presented at INTERSPEECH 2020, kept the band-and-pitch design but reported higher quality while still using less than 5 percent of a CPU core.

The limitations follow from the design. A band gain cannot separate a voice from noise that occupies the same band at the same moment, so strong noise leaves a muffled or watery voice, a distortion often called musical noise. RNNoise is trained to separate speech from background sounds, not one talker from another, so a competing voice in the background is treated as speech. Its quality is bounded by the training mixtures, and a 2017-era model can misjudge noise types it never heard. The 10 millisecond look-ahead keeps latency low for calls but limits how much future context the network can use.

Using RNNoise in practice requires little. The repository builds with a standard configure and make process, downloads the model files during setup, and includes a demo that reads headerless 16-bit mono audio at 48 kHz. Developers who want a custom model can generate features from their own speech and noise files with the provided tools, train with the included Python script and compile the quantized weights back into the library. The README recommends at least 10,000 training sequences and preferably 200,000 or more.