HejuruUbuyobozi bukurikira
Kumenyekanisha Amarangamutima
AI y'amajwi
Ubuyobozi
Kumenyekanisha imvugo (VSR), cyangwa gusoma iminwa ya AI, byandukura imvugo ivuye mumashusho yumunwa wumuvugizi udakoresheje amajwi.
Umva iki gitaboAI-byavuzwe · ~8 min
Kuva mu mwaka wa 2016, sisitemu yo kwiga yimbitse yatojwe kumasaha ibihumbi ya videwo yavuye mumagambo mato mato ajyanye no kwandikirana amagambo, hamwe n’ikosa ry’amagambo hafi 19% ku biganiro bya TED mu 2023. Bifite akamaro ku bantu badashobora kuvuga amajwi, ahantu h’urusaku, no kujya impaka ku buzima bwite.
Visual speech recognition is the task of turning silent video of a person talking into text. Humans call it lip reading, though the face carries more than lips: jaw, teeth and tongue are partly visible too. The machine version takes a sequence of video frames, usually 25 per second, cropped around the mouth, and outputs words. A closely related task, audio-visual speech recognition (AVSR), uses both the sound and the video, which helps most when audio is noisy.
Lip reading is fundamentally ambiguous. A phoneme is the smallest unit of speech sound, and a viseme is its visual counterpart, the mouth shape that goes with it. Many phonemes share one viseme; the Oxford "Lip Reading Sentences in the Wild" paper uses p and b as an example of sounds that produce the same lip sequence. The LipNet paper cites a 1968 study that grouped 23 initial consonant phonemes into just five visually confusable categories. Context resolves much of the ambiguity, which is why sentence-level models with language models do far better than word-by-word guessing. Human performance is low: the LipNet authors note literature estimates of around 20% accuracy for human lip readers on general speech.
A modern system works in four steps. First, a face detector and landmark tracker find the mouth in every frame, and the crop is stabilized and resized to a fixed size, such as 128 by 128 pixels in DeepMind's 2018 pipeline. Second, a visual front end, usually a 3D convolutional network that looks at several frames at once, extracts features describing lip motion. Third, a sequence model, earlier a recurrent network and now usually a Transformer or Conformer, relates those features across the whole utterance. Fourth, a decoder turns the sequence into text, either directly as characters using connectionist temporal classification (CTC, a loss that lets the network output a label per frame without knowing exact alignments) or attention, or as phonemes that a separate decoder maps to words with a pronunciation lexicon and a language model.
The field moved fast after 2016. LipNet, by Yannis Assael, Brendan Shillingford, Shimon Whiteson and Nando de Freitas, was the first end-to-end sentence-level lip reader. It was tested on GRID, a constrained dataset of five-word commands such as "place red at C zero again", where it reached 95.2% sentence accuracy and a 4.8% word error rate on speakers seen in training. That same month, Joon Son Chung, Andrew Senior, Oriol Vinyals and Andrew Zisserman released "Lip Reading Sentences in the Wild", which built the Lip Reading Sentences dataset of over 100,000 natural sentences from BBC programs recorded between 2010 and 2016, and showed the model outperforming a professional lip reader. In 2018, the Oxford group released LRS3, over 400 hours of TED and TEDx talks, which became the main public benchmark. The same year, DeepMind and Google researchers scaled to 3,886 hours from YouTube with a vocabulary of 127,055 words, reaching a 40.9% word error rate where professional lip readers scored 86.4% or 92.9%.
The next leap came from self-supervised learning, which learns from unlabeled data first. Meta's AV-HuBERT (Bowen Shi and colleagues, ICLR 2022) masks parts of audio-visual video and learns to predict clusters of hidden features, then fine-tunes on a small labeled set. With 30 hours of labels it reached 32.5% word error rate on LRS3, beating a previous system trained on about 31,000 hours of transcribed video, and 26.9% with 433 labeled hours plus self-training. In 2023, Auto-AVSR from Imperial College London and Meta used public speech recognizers such as Whisper to automatically transcribe about 2,600 hours of English video from VoxCeleb2 and AVSpeech, training on 3,448 hours in total. Its visual-only model reached 19.1% word error rate on LRS3.
Compared with audio speech recognition, lip reading remains far less accurate. The same Auto-AVSR paper reports audio-visual error of 0.9% on LRS3, roughly 20 times lower than its visual-only result. Video therefore adds the most value as a complement: the 2017 Oxford work showed visual information improves recognition even when audio is available, and AVSR systems hold up better in loud environments. Compared with human lip readers, the best systems are clearly better on benchmark video, but those comparisons use frontal, well-lit footage of the type the models were trained on.
Limitations are significant. The DeepMind authors state plainly that their model only performs well on high-quality video of a subject facing the camera at a certain distance, and not in other contexts. Side angles, low resolution, occlusion by hands or masks, beards, accents, and fast speech all raise error rates. Most large datasets are English and drawn from broadcast or online talks, so performance on other languages and everyday speech is weaker. Unseen speakers are harder: LipNet's word error rate rose from 4.8% to 11.4% on speakers outside training. Building datasets is costly; in DeepMind's YouTube pipeline only about 2% of candidate clips passed the filters.
The technology also raises privacy concerns. The LipNet authors list security and silent-movie processing among potential uses, and a system that reads speech from video could in principle be applied to surveillance footage without a speaker's knowledge, although current accuracy on distant, angled, low-resolution video is poor. On the assistive side, the DeepMind paper notes that 103,925 tracheostomies were performed in the United States in 2014, a procedure that can leave patients unable to voice speech, and frames silent speech recognition as a potential communication aid. Training state-of-the-art models takes substantial compute: the 2017 Oxford model trained for about 10 days, and newer systems use thousands of hours of video.
04Urugero rwakazi
Fata videwo y'amasegonda abiri yumuntu uvuga ngo 'shyira umutuku kuri C zeru', yanditswe kuri frame 25 kumasegonda, utange amakaramu 50.
Koresha isura yerekana ibimenyetso kuri buri kintu hanyuma uhinge akarere gafite ibara ryimeza ryerekanwe kumunwa.
Hisha ibihingwa 50 unyuze imbere ya 3D yemeza imbere isohora icyerekezo kimwe kumurongo.
Kugaburira ibice 50 muri Transformer cyangwa kodegisi isubirwamo kugirango buri cyiciro kigaragaza imvugo yose.
Kwemeza hamwe na CTC kumurongo ushakisha wongeyeho icyitegererezo cyururimi, gisenyuka inshuro nyinshi ibisubizo hamwe nibisobanuro mumiterere cyangwa fonema ikurikirana.
Gereranya ibisohoka ninteruro yukuri hanyuma ubare igipimo cyikosa ryijambo nkibisimbuza wongeyeho gusiba wongeyeho ibyongeweho bigabanijwe numubare wamagambo.
Ibyo byerekana
Ibisohoka ni inyandiko ikeka nka 'shyira umutuku kuri C zeru'. Niba icyitegererezo gisubije 'uburiri bwumwanya kuri C zeru', ikosa nimwe risimburwa mumagambo atanu, igipimo cyamagambo 20%, kandi ryerekana p, b na m viseme idasobanutse neza imiterere nururimi bigamije gukemura.
LipNet yo muri kaminuza ya Oxford na DeepMind (2016) yageze kuri 95.2% byukuri kuri interuro kuri GRID corpus yinteruro ngufi zisa n’amategeko, ugereranije ni 52.3% kubasomyi batatu bafite ubumuga bwo kutumva.
Oxford's Watch, Umva, Kwitabira no Kwandika (CVPR 2017), yatojwe kuri tereviziyo ya BBC, yandukura amagambo agera kuri kimwe cya kabiri mu magambo y’ibizamini, mu gihe umusomyi w’iminwa wabigize umwuga ufite uburambe bwimyaka 10 yabonye munsi ya kimwe cya kane.
DeepMind na Google bubatse dataset yamasaha 3.886 yo gusoma iminwa yo muri YouTube muri 2018 kandi batangaza ko igipimo cyamakosa cyamagambo 40.9%, biterwa nabarwayi babuze ijwi nyuma ya tracheostomy.
AV-HuBERT ya Meta (ICLR 2022) yigiye kuri videwo yerekana amajwi n'amashusho kandi itagera kuri 32.5% igipimo cyo gusoma umunwa kuri LRS3 ukoresheje amasaha 30 gusa yamakuru yanditse.
Kumenyekanisha amajwi-amashusho yerekana guhuza iminwa n'amajwi; sisitemu ya Auto-AVSR (ICASSP 2023) yageze kuri 0.9% igipimo cyamakosa yijambo kurwego rwa LRS3 TED.
The clearest trend in published work is that more data, even with imperfect labels, keeps improving visual speech recognition. The Auto-AVSR authors (2023) reported that visual-only error kept falling as automatically transcribed hours were added, while audio-only error saturated beyond about 1,500 hours. That suggests larger, more diverse video collections and better self-supervised pretraining, as in AV-HuBERT (2022), will drive further gains. Open problems include robustness to real-world video with side views, low light and low resolution, which the 2018 DeepMind authors flagged as a hard limit of their system; coverage of languages beyond English; and handling speakers with atypical articulation, who are often the people who would benefit most from assistive lip reading. Benchmarks such as LRS3 consist of TED talks, so results there may overstate performance on everyday conversation. The social questions are as open as the technical ones. Using lip reading on video recorded without consent would raise privacy and civil-liberties issues, and dataset construction from public online video raises consent questions of its own. Whether assistive products for people with voice loss reach routine clinical use depends on accuracy for individual patients rather than benchmark averages.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Kumenyekanisha imvugo (VSR), cyangwa gusoma iminwa ya AI, byandukura imvugo ivuye mumashusho yumunwa wumuvugizi udakoresheje amajwi. Kuva mu mwaka wa 2016, sisitemu yo kwiga yimbitse yatojwe kumasaha ibihumbi ya videwo yavuye mumagambo mato mato ajyanye no kwandikirana amagambo, hamwe n’ikosa ry’amagambo hafi 19% ku biganiro bya TED mu 2023. Bifite akamaro ku bantu badashobora kuvuga amajwi, ahantu h’urusaku, no kujya impaka ku buzima bwite.
Amatsinda nka p, b, na m yerekana impamvu igaragara neza yonyine idashobora gutandukanya foneme.
Uburyo burashobora kuzuzanya, hamwe ninyungu bitewe nubushakashatsi hamwe nicyitegererezo.
3D imyanzuro itunganya ama frame menshi hamwe, ifata uburyo umunwa ugenda mugihe gito aho kuba ishusho imwe ihamye.
Kugaragara no gufata ubuziranenge bigira ingaruka kumunwa amakuru aboneka kubitekerezo.
Ibipimo ngenderwaho ntabwo bishyiraho imikorere yisi yose mundimi, abavuga, hamwe nuburyo bwo gufata.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Kumenyekanisha Amarangamutima
AI y'amajwi