應用指南

人工智慧在唇讀和視覺語音辨識的應用

視覺語音辨識 (VSR) 或 AI 唇讀可以在不使用音訊的情況下從說話者嘴部的視訊中轉錄語音。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 透過唇讀管道追蹤兩秒剪輯
  5. 現實世界的實施
  6. 人工智慧在唇讀和視覺語音辨識領域的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

自 2016 年以來,經過數千小時視訊訓練的深度學習系統已從小型固定詞彙發展到開放詞彙轉錄,到 2023 年 TED 演講的單字錯誤率接近 19%。這對於無法語音講話的人、吵雜的環境和隱私辯論都很重要。

重點摘要

  1. 視覺語音辨識從臉部無聲視訊轉錄語音,而視聽語音辨識則將視訊與音訊結合起來,以提高噪音中的穩健性。
  2. 唇讀本質上是模棱兩可的,因為 p 和 b 等不同的聲音在嘴唇上看起來是相同的,因此模型在很大程度上依賴於句子上下文。
  3. LipNet (2016) 是第一個端對端句子級唇語閱讀器,在受限 GRID 語料庫上達到 95.2% 的句子準確率。
  4. 在 LRS3 TED 演講基準測試中,Auto-AVSR (2023) 的最佳純視覺錯誤率降至 19.1%,而其視聽模型達到 0.9%。
  5. 已發布的唇讀系統在正面、光線充足、高品質的影片上效果最佳,但在側面角度、低解析度和看不見的揚聲器上表現會急劇下降。

深入探討

Visual speech recognition is the task of turning silent video of a person talking into text. Humans call it lip reading, though the face carries more than lips: jaw, teeth and tongue are partly visible too. The machine version takes a sequence of video frames, usually 25 per second, cropped around the mouth, and outputs words. A closely related task, audio-visual speech recognition (AVSR), uses both the sound and the video, which helps most when audio is noisy.

Lip reading is fundamentally ambiguous. A phoneme is the smallest unit of speech sound, and a viseme is its visual counterpart, the mouth shape that goes with it. Many phonemes share one viseme; the Oxford "Lip Reading Sentences in the Wild" paper uses p and b as an example of sounds that produce the same lip sequence. The LipNet paper cites a 1968 study that grouped 23 initial consonant phonemes into just five visually confusable categories. Context resolves much of the ambiguity, which is why sentence-level models with language models do far better than word-by-word guessing. Human performance is low: the LipNet authors note literature estimates of around 20% accuracy for human lip readers on general speech.

A modern system works in four steps. First, a face detector and landmark tracker find the mouth in every frame, and the crop is stabilized and resized to a fixed size, such as 128 by 128 pixels in DeepMind's 2018 pipeline. Second, a visual front end, usually a 3D convolutional network that looks at several frames at once, extracts features describing lip motion. Third, a sequence model, earlier a recurrent network and now usually a Transformer or Conformer, relates those features across the whole utterance. Fourth, a decoder turns the sequence into text, either directly as characters using connectionist temporal classification (CTC, a loss that lets the network output a label per frame without knowing exact alignments) or attention, or as phonemes that a separate decoder maps to words with a pronunciation lexicon and a language model.

The field moved fast after 2016. LipNet, by Yannis Assael, Brendan Shillingford, Shimon Whiteson and Nando de Freitas, was the first end-to-end sentence-level lip reader. It was tested on GRID, a constrained dataset of five-word commands such as "place red at C zero again", where it reached 95.2% sentence accuracy and a 4.8% word error rate on speakers seen in training. That same month, Joon Son Chung, Andrew Senior, Oriol Vinyals and Andrew Zisserman released "Lip Reading Sentences in the Wild", which built the Lip Reading Sentences dataset of over 100,000 natural sentences from BBC programs recorded between 2010 and 2016, and showed the model outperforming a professional lip reader. In 2018, the Oxford group released LRS3, over 400 hours of TED and TEDx talks, which became the main public benchmark. The same year, DeepMind and Google researchers scaled to 3,886 hours from YouTube with a vocabulary of 127,055 words, reaching a 40.9% word error rate where professional lip readers scored 86.4% or 92.9%.

The next leap came from self-supervised learning, which learns from unlabeled data first. Meta's AV-HuBERT (Bowen Shi and colleagues, ICLR 2022) masks parts of audio-visual video and learns to predict clusters of hidden features, then fine-tunes on a small labeled set. With 30 hours of labels it reached 32.5% word error rate on LRS3, beating a previous system trained on about 31,000 hours of transcribed video, and 26.9% with 433 labeled hours plus self-training. In 2023, Auto-AVSR from Imperial College London and Meta used public speech recognizers such as Whisper to automatically transcribe about 2,600 hours of English video from VoxCeleb2 and AVSpeech, training on 3,448 hours in total. Its visual-only model reached 19.1% word error rate on LRS3.

Compared with audio speech recognition, lip reading remains far less accurate. The same Auto-AVSR paper reports audio-visual error of 0.9% on LRS3, roughly 20 times lower than its visual-only result. Video therefore adds the most value as a complement: the 2017 Oxford work showed visual information improves recognition even when audio is available, and AVSR systems hold up better in loud environments. Compared with human lip readers, the best systems are clearly better on benchmark video, but those comparisons use frontal, well-lit footage of the type the models were trained on.

Limitations are significant. The DeepMind authors state plainly that their model only performs well on high-quality video of a subject facing the camera at a certain distance, and not in other contexts. Side angles, low resolution, occlusion by hands or masks, beards, accents, and fast speech all raise error rates. Most large datasets are English and drawn from broadcast or online talks, so performance on other languages and everyday speech is weaker. Unseen speakers are harder: LipNet's word error rate rose from 4.8% to 11.4% on speakers outside training. Building datasets is costly; in DeepMind's YouTube pipeline only about 2% of candidate clips passed the filters.

The technology also raises privacy concerns. The LipNet authors list security and silent-movie processing among potential uses, and a system that reads speech from video could in principle be applied to surveillance footage without a speaker's knowledge, although current accuracy on distant, angled, low-resolution video is poor. On the assistive side, the DeepMind paper notes that 103,925 tracheostomies were performed in the United States in 2014, a procedure that can leave patients unable to voice speech, and frames silent speech recognition as a potential communication aid. Training state-of-the-art models takes substantial compute: the 2017 Oxford model trained for about 10 days, and newer systems use thousands of hours of video.

04工作範例

透過唇讀管道追蹤兩秒剪輯

  1. 拍攝一個兩秒的正面視頻,其中有人說“將紅色放在 C 零處”,以每秒 25 幀的速度錄製,總共 50 幀。

  2. 在每個畫面上運行臉部標誌偵測器,並裁剪以嘴部為中心的固定大小的灰階區域。

  3. 將 50 個裁切影像通過 3D 卷積前端,該前端每幀輸出一個特徵向量。

  4. 將 50 個向量輸入 Transformer 或循環編碼器,以便每個幀的表示反映整個話語。

  5. 使用 CTC 波束搜尋和語言模型進行解碼,將重複的輸出和空白折疊成字元或音素序列。

  6. 將輸出與真實句子進行比較,並計算單字錯誤率:取代加刪除加插入除以參考單字的數量。

它顯示了什麼

輸出是文字猜測,例如“將紅色置於 C 零處”。如果模型返回“將床放在 C 零處”,則錯誤是五個單字中的一個替換,即 20% 的單字錯誤率,並且它說明了上下文和語言模型旨在解決的 p、b 和 m 視位歧義。

現實世界的實施

牛津大學的 LipNet 和 DeepMind (2016) 在 GRID 語料庫上的類似命令的短句句子準確率達到了 95.2%,而三個聽力受損的唇讀者的平均準確率為 52.3%。

牛津大學的觀看、聆聽、參加和拼寫模型 (CVPR 2017) 在 BBC 電視上進行了訓練,轉錄了測試剪輯中大約一半的單詞,而擁有約 10 年經驗的專業唇語閱讀者只轉錄了不到四分之一的單詞。

DeepMind 和 Google 於 2018 年根據 YouTube 建立了一個 3,886 小時的唇讀資料集,並報告了 40.9% 的單字錯誤率,部分原因是氣管切開術後失聲的患者。

Meta的AV-HuBERT(ICLR 2022)從未標記的視聽影片中學習,僅使用30小時的標記資料就在LRS3上達到了32.5%的唇讀錯誤率。

視聽語音辨識將嘴唇運動與音訊結合; Auto-AVSR 系統 (ICASSP 2023) 在 LRS3 TED 演講基準測試中達到了 0.9% 的單字錯誤率。

人工智慧在唇讀和視覺語音辨識領域的未來

The clearest trend in published work is that more data, even with imperfect labels, keeps improving visual speech recognition. The Auto-AVSR authors (2023) reported that visual-only error kept falling as automatically transcribed hours were added, while audio-only error saturated beyond about 1,500 hours. That suggests larger, more diverse video collections and better self-supervised pretraining, as in AV-HuBERT (2022), will drive further gains. Open problems include robustness to real-world video with side views, low light and low resolution, which the 2018 DeepMind authors flagged as a hard limit of their system; coverage of languages beyond English; and handling speakers with atypical articulation, who are often the people who would benefit most from assistive lip reading. Benchmarks such as LRS3 consist of TED talks, so results there may overstate performance on everyday conversation. The social questions are as open as the technical ones. Using lip reading on video recorded without consent would raise privacy and civil-liberties issues, and dataset construction from public online video raises consent questions of its own. Whether assistive products for people with voice loss reach routine clinical use depends on accuracy for individual patients rather than benchmark averages.

付諸實踐

為什麼這很重要

  • 配裝選擇. 應用級設計決定了人工智慧是否能改善實際結果。
  • 團隊與工作流程. 良好的工作流程整合可以創造使用者值得信賴的生產力效益。
  • 風險與安全. 範圍明確的用例可以減少變更疲勞和實施風險。

留意

  • 將損壞的流程自動化可能會加劇現有問題。
  • 團隊可能會過度自動化並消除所需的人工判斷。
  • 如果不持續評估輸出,品質可能會出現偏差。

後續步驟

  1. 繪製目前工作流程並確定摩擦最大的步驟。
  2. 在完全自動化之前定義人工檢查點。
  3. 對使用者進行提示、升級路徑和品質標準的訓練。
  4. 追蹤任務級結果以確認持續價值。

資料來源與延伸閱讀

  1. arXivLipNet:端到端句子級唇讀
  2. arXiv (CVPR 2017)野外唇讀句子
  3. arXivLRS3-TED:用於視覺語音辨識的大規模資料集
  4. arXiv大規模視覺語音識別
  5. arXiv (ICLR 2022)透過掩碼多模態聚類預測學習視聽語音表示 (AV-HuBERT)
  6. arXiv (ICASSP 2023)Auto-AVSR:具有自動標籤的視聽語音識別

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加唇讀和視覺語音辨識中的人工智慧測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是唇讀和視覺語音辨識中的人工智慧?

視覺語音辨識 (VSR) 或 AI 唇讀可以在不使用音訊的情況下從說話者嘴部的視訊中轉錄語音。自 2016 年以來,經過數千小時視訊訓練的深度學習系統已從小型固定詞彙發展到開放詞彙轉錄,到 2023 年 TED 演講的單字錯誤率接近 19%。這對於無法語音講話的人、吵雜的環境和隱私辯論都很重要。

視位分組在視覺語音辨識中描述什麼歧義?

p、b 和 m 等組別說明了為什麼僅靠可見發音可能無法區分每個音素。

當噪音增加時,視覺流如何幫助視聽辨識器?

這些模式可以相互補充,其益處取決於觀察結果和模型。

唇讀模型中的 3D 卷積前端有助於捕捉什麼?

3D 卷積同時處理多個幀,捕捉嘴巴在短時間內的移動情況,而不是單一靜態影像。

哪些擷取條件可以隱藏視覺語音模型所需的資訊?

可見性和捕獲品質影響可用於推理的口腔資訊。

原始LipNet句子辨識結果該附加什麼評估範圍?

基準測試結果並不能確定跨語言、說話者和捕捉條件的通用表現。