Audio AI-GIDS

HuBERT zelfgecontroleerde toespraak

HuBERT (Hidden-Unit BERT) is a self-supervised speech model from Facebook AI, published in June 2021, that learns from unlabeled audio by predicting cluster labels for masked stretches of speech.

Luister naar deze gidsAI-verteld · ~8 min

Op deze pagina9 minuten lezen
  1. Overzicht
  2. Belangrijkste inzichten
  3. Diepe duik
  4. Making HuBERT units from a speech clip
  5. Implementatie in de echte wereld
  6. De toekomst van HuBERT zelfgecontroleerde spraak
  7. Het in de praktijk brengen
  8. Bronnen en verdere literatuur
  9. Blijf verkennen
  10. Veelgestelde vragen

Overzicht

It matched or beat wav2vec 2.0 on LibriSpeech speech recognition, and its discrete units became a standard building block for speech generation, compression and voice research.

Belangrijkste inzichten

  1. HuBERT learns speech representations by predicting k-means cluster labels for masked spans of unlabeled audio.
  2. Its first labels come from clustering MFCC features into 100 groups, then from clustering its own hidden layers into 500.
  3. Fine-tuned on only 10 minutes of transcripts, HuBERT X-LARGE reached 4.6% WER on test-clean and 6.8% on test-other.
  4. HuBERT computes its loss only on masked frames, forcing it to infer hidden speech from surrounding context.
  5. HuBERT's discrete units are widely used for textless speech generation and low-bitrate speech resynthesis.

Diepe duik

HuBERT, short for Hidden-Unit BERT, is a method for learning useful representations of speech from recordings that have no transcripts. It was introduced by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov and Abdelrahman Mohamed in a paper posted on June 14, 2021, with a Facebook AI blog post the next day. The trained model turns raw audio into a sequence of vectors, one every 20 milliseconds, that capture the sounds being spoken. Those vectors can then be fine-tuned with a small amount of labeled data for speech recognition or used directly for other speech tasks.

The idea borrows from BERT, the text model that learns by hiding some words in a sentence and predicting them. Speech makes this harder in three ways that the paper names: each utterance contains many sound units, there is no dictionary of those units to predict, and units have different lengths with no marked boundaries. HuBERT's answer is to create its own targets with an offline clustering step. K-means clustering, which groups similar vectors into a fixed number of clusters, assigns every 20 ms frame a cluster number. These numbers are the hidden units. They are noisy stand-ins for phonetic categories, but they are consistent: similar sounds get similar labels.

Training then proceeds like BERT. A convolutional encoder turns 16 kHz audio into frames, downsampling by a factor of 320. About 8% of frames are chosen as starting points and spans of 10 frames from each are replaced by a learned mask embedding. A transformer encoder reads the partly masked sequence and, at every masked frame, predicts which cluster that frame was assigned. The key design choice is computing the loss only on masked frames. To predict a hidden frame's label, the model must infer it from surrounding context, which forces it to learn both what sounds are present, like an acoustic model, and how speech sequences are structured, like a language model.

The clustering improves over iterations. In the first round, the authors clustered 39-dimensional MFCC features, a classic hand-designed summary of the speech spectrum, into 100 clusters. After training the BASE model for 250,000 steps on LibriSpeech's 960 hours, they clustered the output of its sixth transformer layer into 500 clusters and trained again for 400,000 steps. The LARGE and X-LARGE models were trained for one iteration on the 60,000-hour Libri-Light collection, using labels from the ninth layer of the second-iteration BASE model. The paper notes that HuBERT depends more on the consistency of the clustering than on how accurate the cluster labels are as phonetic categories.

The results matched or exceeded wav2vec 2.0, the leading self-supervised speech model at the time, across fine-tuning sets of 10 minutes, 1 hour, 10 hours, 100 hours and 960 hours. With only 10 minutes of labels, HuBERT LARGE reached 4.7% word error rate on test-clean and 7.6% on test-other, and the 1-billion-parameter X-LARGE reached 4.6% and 6.8%. With all 960 hours of labels, X-LARGE reached 1.8% on test-clean and 2.9% on test-other, against 1.8% and 3.3% for wav2vec 2.0 LARGE in the paper's comparison. The authors summarize the gain as up to 19% and 13% relative word error rate reduction on dev-other and test-other. The exceptions were small: with 100 hours of labels, HuBERT LARGE and BASE were each 0.1 points behind wav2vec 2.0 on one test set.

The main alternative, wav2vec 2.0, also masks speech, but it learns its discrete targets during training and uses a contrastive loss that must pick the true frame from sampled distractors. The HuBERT paper points out that this requires careful choice of negative samples, an extra diversity loss and a temperature schedule. HuBERT separates the two problems, discovering units offline and then training with a simple cross-entropy loss, which made training more direct. Compared with supervised speech recognition, both approaches need far less labeled data, which matters for languages and domains where transcripts are scarce.

HuBERT has clear limits. The published models were trained only on English read audiobooks from LibriVox, so they reflect that style and language, and results on other languages or spontaneous, noisy speech must be measured separately. The method needs repeated rounds of clustering and pre-training, and k-means over high-dimensional features for hundreds of hours of audio requires subsampling to fit in memory, which the authors note. HuBERT is an encoder, not a complete transcription system: it needs fine-tuning and a decoder, and the strongest reported numbers also used a language model during decoding. Its units capture mainly what is said, which is useful for content but means speaker identity and prosody must be handled by other components in generation systems.

The practical cost is concentrated in pre-training. The authors trained BASE on 32 GPUs and the LARGE and X-LARGE models on 128 and 256 GPUs for 400,000 steps each. Fine-tuning and use are far cheaper: pretrained and fine-tuned checkpoints were released through Facebook's fairseq repository and are available in Hugging Face Transformers, where a model loads in a few lines and expects raw audio sampled at 16 kHz. Three sizes exist, BASE with 95 million parameters, LARGE with 317 million and X-LARGE with 964 million.

04Uitgewerkt voorbeeld

Making HuBERT units from a speech clip

  1. Load a 10-second English speech clip and resample it to 16 kHz, giving 160,000 samples.

  2. Pass it through a pretrained HuBERT BASE model and collect the hidden states, about 500 frames of 768 numbers each.

  3. Take the output of a middle transformer layer and fit k-means with 100 clusters on frames from many clips.

  4. Assign every frame of the test clip to its nearest cluster, giving a sequence of 500 integers.

  5. Merge consecutive repeated integers to get a shorter unit sequence, as textless speech models often do.

  6. Compare unit sequences for two speakers saying the same sentence and check which units line up.

Wat het laat zien

The learner gets a discrete pseudo-text for speech, one unit per 20 ms before merging. Similar unit sequences for the same words from different speakers show that HuBERT units mainly capture content, which is why they serve as tokens for spoken language models.

Implementatie in de echte wereld

Low-resource speech recognition: after pre-training on 60,000 hours of unlabeled audiobook speech, HuBERT LARGE fine-tuned on only 10 minutes of transcribed audio reached 4.7% word error rate on LibriSpeech test-clean and 7.6% on test-other.

Textless speech generation: Meta's Generative Spoken Language Modeling work (2021) turned HuBERT features into discrete pseudo-text units, trained a language model on them and synthesized speech without any text.

Very low bitrate compression: Meta reported that its companion speech resynthesis work using HuBERT units reached 365 bits per second, and HuBERT ranked second only to uncompressed audio in a MUSHRA listening test.

Off-the-shelf transcription: Hugging Face Transformers hosts HuBERT checkpoints such as facebook/hubert-large-ls960-ft that transcribe 16 kHz English audio through a standard speech recognition pipeline.

Audio classification: the same library provides HubertForSequenceClassification, a HuBERT encoder with a pooled classification head, for tasks such as keyword spotting from the SUPERB benchmark.

De toekomst van HuBERT zelfgecontroleerde spraak

HuBERT's most lasting influence may be outside speech recognition. Its discrete units showed that speech can be modeled like text without any transcripts: Meta's Generative Spoken Language Modeling paper (February 2021, revised September 2021) compared HuBERT units with CPC and wav2vec 2.0 units for training language models on raw audio, and Meta's June 2021 announcement described generation and very low bitrate resynthesis built on HuBERT. Open questions include how to extend self-supervised speech models to many languages, accents and recording conditions when public pre-training data is dominated by English audiobooks, how to capture prosody, emotion and speaker identity that clustered content units tend to discard, and how to reduce the cost of iterative clustering and pre-training at large scale. The HuBERT authors said in 2021 that they planned to simplify training into a single phase instead of repeated rounds of clustering and pre-training. Comparisons also keep shifting. Newer models trained on far more varied data, with or without labels, have changed what counts as state of the art on many benchmarks since 2021, so HuBERT is now often used as a well-understood baseline and feature extractor rather than the best available recognizer. Choosing it today is mainly a question of whether its open checkpoints, 16 kHz English training data and discrete units fit the task.

Het in de praktijk brengen

Waarom het ertoe doet

  • Toegang en bereik. Het verbetert de toegankelijkheid via transcriptie, gesproken tekst en spraakinterfaces.
  • Kosten en budget. Mediateams kunnen met kleinere budgetten sneller gepolijste audio leveren.
  • Snelheid en schaal. Klantgerichte systemen kunnen gesproken interacties op grotere schaal verwerken.

Pas op voor

  • Het risico op stemmisbruik en imitatie neemt toe als de toestemming ontbreekt.
  • De nauwkeurigheid kan afnemen bij accenten, dialecten of luidruchtige omgevingen.
  • Synthetische audio kan worden aangezien voor authentieke spraak zonder duidelijke labels.

Volgende stappen

  1. Verkrijg expliciete toestemming voor het vastleggen, klonen en hergebruiken van spraak.
  2. Test de kwaliteit van diverse sprekers en achtergrondomstandigheden.
  3. Bepaal wanneer een mens de output moet beoordelen of goedkeuren.
  4. Label synthetische audio en houd de herkomstgegevens bij voor verantwoording.

Bronnen en verdere literatuur

  1. arXivHuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
  2. Meta AIHuBERT: Self-supervised representation learning for speech recognition, generation, and compression
  3. arXivGenerative Spoken Language Modeling from Raw Audio
  4. Hugging Face Transformers documentationHuBERT

Blijf verkennen

Gratis nieuwsbrief

Ontvang de dagelijkse AI-briefing

Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.

Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.

Test jezelf

Take the HuBERT Self-Supervised Speech quiz

Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.

Quiz starten

Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie

Veelgestelde vragen

What is HuBERT Self-Supervised Speech?

HuBERT (Hidden-Unit BERT) is a self-supervised speech model from Facebook AI, published in June 2021, that learns from unlabeled audio by predicting cluster labels for masked stretches of speech. It matched or beat wav2vec 2.0 on LibriSpeech speech recognition, and its discrete units became a standard building block for speech generation, compression and voice research.

Hoe genereert HuBERT de doelen die het tijdens de training probeert te voorspellen?

HuBERT clustert audioframekenmerken (via k-middelen) in discrete verborgen eenheden en voorspelt die clusterlabels voor gemaskeerde frames.

Welk bekend tekstmodel inspireerde HuBERT's trainingsschema voor gemaskerde voorspellingen?

HuBERT (Hidden-Unit BERT) leent het gemaskeerde voorspellingsdoel van BERT en past het toe op discrete spraakeenheden in plaats van op teksttokens.

Hoe verbetert HuBERT zijn doellabels tijdens opeenvolgende trainingsiteraties?

In later iterations, HuBERT clusters the model’s learned representations to create updated pseudo-labels; the paper does not identify these labels as verified phonemes.

Welke functies zijn doorgaans geclusterd in de allereerste iteratie van HuBERT?

De eerste iteratie clustert basis MFCC-functies; latere iteraties gebruiken rijkere Transformer-laagrepresentaties.

Waarom kan HuBERT een nuttige structuur leren, zelfs als de clustering ervan niet perfect is?

Omdat het doel alleen maar het voorspellen van toegewezen cluster-ID's voor gemaskerde frames is, blijft de taak leerbaar en consistent, ondanks luidruchtige clusters.