HuBERT Selbstüberwachte Rede

HuBERT, short for Hidden-Unit BERT, is a method for learning useful representations of speech from recordings that have no transcripts. It was introduced by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov and Abdelrahman Mohamed in a paper posted on June 14, 2021, with a Facebook AI blog post the next day. The trained model turns raw audio into a sequence of vectors, one every 20 milliseconds, that capture the sounds being spoken. Those vectors can then be fine-tuned with a small amount of labeled data for speech recognition or used directly for other speech tasks.

The idea borrows from BERT, the text model that learns by hiding some words in a sentence and predicting them. Speech makes this harder in three ways that the paper names: each utterance contains many sound units, there is no dictionary of those units to predict, and units have different lengths with no marked boundaries. HuBERT's answer is to create its own targets with an offline clustering step. K-means clustering, which groups similar vectors into a fixed number of clusters, assigns every 20 ms frame a cluster number. These numbers are the hidden units. They are noisy stand-ins for phonetic categories, but they are consistent: similar sounds get similar labels.

Training then proceeds like BERT. A convolutional encoder turns 16 kHz audio into frames, downsampling by a factor of 320. About 8% of frames are chosen as starting points and spans of 10 frames from each are replaced by a learned mask embedding. A transformer encoder reads the partly masked sequence and, at every masked frame, predicts which cluster that frame was assigned. The key design choice is computing the loss only on masked frames. To predict a hidden frame's label, the model must infer it from surrounding context, which forces it to learn both what sounds are present, like an acoustic model, and how speech sequences are structured, like a language model.

The clustering improves over iterations. In the first round, the authors clustered 39-dimensional MFCC features, a classic hand-designed summary of the speech spectrum, into 100 clusters. After training the BASE model for 250,000 steps on LibriSpeech's 960 hours, they clustered the output of its sixth transformer layer into 500 clusters and trained again for 400,000 steps. The LARGE and X-LARGE models were trained for one iteration on the 60,000-hour Libri-Light collection, using labels from the ninth layer of the second-iteration BASE model. The paper notes that HuBERT depends more on the consistency of the clustering than on how accurate the cluster labels are as phonetic categories.

The results matched or exceeded wav2vec 2.0, the leading self-supervised speech model at the time, across fine-tuning sets of 10 minutes, 1 hour, 10 hours, 100 hours and 960 hours. With only 10 minutes of labels, HuBERT LARGE reached 4.7% word error rate on test-clean and 7.6% on test-other, and the 1-billion-parameter X-LARGE reached 4.6% and 6.8%. With all 960 hours of labels, X-LARGE reached 1.8% on test-clean and 2.9% on test-other, against 1.8% and 3.3% for wav2vec 2.0 LARGE in the paper's comparison. The authors summarize the gain as up to 19% and 13% relative word error rate reduction on dev-other and test-other. The exceptions were small: with 100 hours of labels, HuBERT LARGE and BASE were each 0.1 points behind wav2vec 2.0 on one test set.

The main alternative, wav2vec 2.0, also masks speech, but it learns its discrete targets during training and uses a contrastive loss that must pick the true frame from sampled distractors. The HuBERT paper points out that this requires careful choice of negative samples, an extra diversity loss and a temperature schedule. HuBERT separates the two problems, discovering units offline and then training with a simple cross-entropy loss, which made training more direct. Compared with supervised speech recognition, both approaches need far less labeled data, which matters for languages and domains where transcripts are scarce.

HuBERT has clear limits. The published models were trained only on English read audiobooks from LibriVox, so they reflect that style and language, and results on other languages or spontaneous, noisy speech must be measured separately. The method needs repeated rounds of clustering and pre-training, and k-means over high-dimensional features for hundreds of hours of audio requires subsampling to fit in memory, which the authors note. HuBERT is an encoder, not a complete transcription system: it needs fine-tuning and a decoder, and the strongest reported numbers also used a language model during decoding. Its units capture mainly what is said, which is useful for content but means speaker identity and prosody must be handled by other components in generation systems.

The practical cost is concentrated in pre-training. The authors trained BASE on 32 GPUs and the LARGE and X-LARGE models on 128 and 256 GPUs for 400,000 steps each. Fine-tuning and use are far cheaper: pretrained and fine-tuned checkpoints were released through Facebook's fairseq repository and are available in Hugging Face Transformers, where a model loads in a few lines and expects raw audio sampled at 16 kHz. Three sizes exist, BASE with 95 million parameters, LARGE with 317 million and X-LARGE with 964 million.