人工智能在唇读和视觉语音识别中的应用

Visual speech recognition is the task of turning silent video of a person talking into text. Humans call it lip reading, though the face carries more than lips: jaw, teeth and tongue are partly visible too. The machine version takes a sequence of video frames, usually 25 per second, cropped around the mouth, and outputs words. A closely related task, audio-visual speech recognition (AVSR), uses both the sound and the video, which helps most when audio is noisy.

Lip reading is fundamentally ambiguous. A phoneme is the smallest unit of speech sound, and a viseme is its visual counterpart, the mouth shape that goes with it. Many phonemes share one viseme; the Oxford "Lip Reading Sentences in the Wild" paper uses p and b as an example of sounds that produce the same lip sequence. The LipNet paper cites a 1968 study that grouped 23 initial consonant phonemes into just five visually confusable categories. Context resolves much of the ambiguity, which is why sentence-level models with language models do far better than word-by-word guessing. Human performance is low: the LipNet authors note literature estimates of around 20% accuracy for human lip readers on general speech.

A modern system works in four steps. First, a face detector and landmark tracker find the mouth in every frame, and the crop is stabilized and resized to a fixed size, such as 128 by 128 pixels in DeepMind's 2018 pipeline. Second, a visual front end, usually a 3D convolutional network that looks at several frames at once, extracts features describing lip motion. Third, a sequence model, earlier a recurrent network and now usually a Transformer or Conformer, relates those features across the whole utterance. Fourth, a decoder turns the sequence into text, either directly as characters using connectionist temporal classification (CTC, a loss that lets the network output a label per frame without knowing exact alignments) or attention, or as phonemes that a separate decoder maps to words with a pronunciation lexicon and a language model.

The field moved fast after 2016. LipNet, by Yannis Assael, Brendan Shillingford, Shimon Whiteson and Nando de Freitas, was the first end-to-end sentence-level lip reader. It was tested on GRID, a constrained dataset of five-word commands such as "place red at C zero again", where it reached 95.2% sentence accuracy and a 4.8% word error rate on speakers seen in training. That same month, Joon Son Chung, Andrew Senior, Oriol Vinyals and Andrew Zisserman released "Lip Reading Sentences in the Wild", which built the Lip Reading Sentences dataset of over 100,000 natural sentences from BBC programs recorded between 2010 and 2016, and showed the model outperforming a professional lip reader. In 2018, the Oxford group released LRS3, over 400 hours of TED and TEDx talks, which became the main public benchmark. The same year, DeepMind and Google researchers scaled to 3,886 hours from YouTube with a vocabulary of 127,055 words, reaching a 40.9% word error rate where professional lip readers scored 86.4% or 92.9%.

The next leap came from self-supervised learning, which learns from unlabeled data first. Meta's AV-HuBERT (Bowen Shi and colleagues, ICLR 2022) masks parts of audio-visual video and learns to predict clusters of hidden features, then fine-tunes on a small labeled set. With 30 hours of labels it reached 32.5% word error rate on LRS3, beating a previous system trained on about 31,000 hours of transcribed video, and 26.9% with 433 labeled hours plus self-training. In 2023, Auto-AVSR from Imperial College London and Meta used public speech recognizers such as Whisper to automatically transcribe about 2,600 hours of English video from VoxCeleb2 and AVSpeech, training on 3,448 hours in total. Its visual-only model reached 19.1% word error rate on LRS3.

Compared with audio speech recognition, lip reading remains far less accurate. The same Auto-AVSR paper reports audio-visual error of 0.9% on LRS3, roughly 20 times lower than its visual-only result. Video therefore adds the most value as a complement: the 2017 Oxford work showed visual information improves recognition even when audio is available, and AVSR systems hold up better in loud environments. Compared with human lip readers, the best systems are clearly better on benchmark video, but those comparisons use frontal, well-lit footage of the type the models were trained on.

Limitations are significant. The DeepMind authors state plainly that their model only performs well on high-quality video of a subject facing the camera at a certain distance, and not in other contexts. Side angles, low resolution, occlusion by hands or masks, beards, accents, and fast speech all raise error rates. Most large datasets are English and drawn from broadcast or online talks, so performance on other languages and everyday speech is weaker. Unseen speakers are harder: LipNet's word error rate rose from 4.8% to 11.4% on speakers outside training. Building datasets is costly; in DeepMind's YouTube pipeline only about 2% of candidate clips passed the filters.

The technology also raises privacy concerns. The LipNet authors list security and silent-movie processing among potential uses, and a system that reads speech from video could in principle be applied to surveillance footage without a speaker's knowledge, although current accuracy on distant, angled, low-resolution video is poor. On the assistive side, the DeepMind paper notes that 103,925 tracheostomies were performed in the United States in 2014, a procedure that can leave patients unable to voice speech, and frames silent speech recognition as a potential communication aid. Training state-of-the-art models takes substantial compute: the 2017 Oxford model trained for about 10 days, and newer systems use thousands of hours of video.