Ubuyobozi

AI mu Gusoma Iminwa no Kumenyekanisha Imvugo

Kumenyekanisha imvugo (VSR), cyangwa gusoma iminwa ya AI, byandukura imvugo ivuye mumashusho yumunwa wumuvugizi udakoresheje amajwi.

Umva iki gitaboAI-byavuzwe · ~8 min

Kuriyi page9 min soma
  1. Incamake
  2. Ibyingenzi byingenzi
  3. Kwibira cyane
  4. Gukurikirana clip-amasegonda abiri ukoresheje umuyoboro usoma iminwa
  5. Gushyira mu bikorwa Isi
  6. Kazoza ka AI mugusoma iminwa no kumenyekanisha imvugo
  7. Kubishyira mubikorwa
  8. Inkomoko no gusoma
  9. Komeza Ubushakashatsi
  10. Ibibazo bikunze kubazwa

Incamake

Kuva mu mwaka wa 2016, sisitemu yo kwiga yimbitse yatojwe kumasaha ibihumbi ya videwo yavuye mumagambo mato mato ajyanye no kwandikirana amagambo, hamwe n’ikosa ry’amagambo hafi 19% ku biganiro bya TED mu 2023. Bifite akamaro ku bantu badashobora kuvuga amajwi, ahantu h’urusaku, no kujya impaka ku buzima bwite.

Ibyingenzi byingenzi

  1. Kumenyekanisha imvugo igaragara byerekana imvugo ivuye muri videwo ituje yo mu maso, mu gihe amajwi-yerekana imvugo ihuza amajwi n'amajwi kugirango bikomere mu rusaku.
  2. Gusoma iminwa ntibisobanutse neza kuko amajwi atandukanye nka p na b asa nkiminwa, bityo moderi zishingiye cyane kumirongo yinteruro.
  3. LipNet (2016) niyo yambere yarangije-kurangiza-interuro-urwego rwumusomyi wiminwa kandi yageze kuri 95.2% byukuri byinteruro kuri GRID corpus yabujijwe.
  4. Ku gipimo cyo kuganira cya LRS3 TED, ikosa ryiza ryerekanwa-ryonyine ryaragabanutse kugera kuri 19.1% muri Auto-AVSR (2023), mugihe moderi yayo yerekana amashusho yageze kuri 0.9%.
  5. Sisitemu yo gusoma iminwa yatangajwe ikora neza imbere, yaka cyane, videwo yo mu rwego rwo hejuru kandi igatesha agaciro cyane kuruhande, imiterere mike hamwe nabavuga batagaragara.

Kwibira cyane

Visual speech recognition is the task of turning silent video of a person talking into text. Humans call it lip reading, though the face carries more than lips: jaw, teeth and tongue are partly visible too. The machine version takes a sequence of video frames, usually 25 per second, cropped around the mouth, and outputs words. A closely related task, audio-visual speech recognition (AVSR), uses both the sound and the video, which helps most when audio is noisy.

Lip reading is fundamentally ambiguous. A phoneme is the smallest unit of speech sound, and a viseme is its visual counterpart, the mouth shape that goes with it. Many phonemes share one viseme; the Oxford "Lip Reading Sentences in the Wild" paper uses p and b as an example of sounds that produce the same lip sequence. The LipNet paper cites a 1968 study that grouped 23 initial consonant phonemes into just five visually confusable categories. Context resolves much of the ambiguity, which is why sentence-level models with language models do far better than word-by-word guessing. Human performance is low: the LipNet authors note literature estimates of around 20% accuracy for human lip readers on general speech.

A modern system works in four steps. First, a face detector and landmark tracker find the mouth in every frame, and the crop is stabilized and resized to a fixed size, such as 128 by 128 pixels in DeepMind's 2018 pipeline. Second, a visual front end, usually a 3D convolutional network that looks at several frames at once, extracts features describing lip motion. Third, a sequence model, earlier a recurrent network and now usually a Transformer or Conformer, relates those features across the whole utterance. Fourth, a decoder turns the sequence into text, either directly as characters using connectionist temporal classification (CTC, a loss that lets the network output a label per frame without knowing exact alignments) or attention, or as phonemes that a separate decoder maps to words with a pronunciation lexicon and a language model.

The field moved fast after 2016. LipNet, by Yannis Assael, Brendan Shillingford, Shimon Whiteson and Nando de Freitas, was the first end-to-end sentence-level lip reader. It was tested on GRID, a constrained dataset of five-word commands such as "place red at C zero again", where it reached 95.2% sentence accuracy and a 4.8% word error rate on speakers seen in training. That same month, Joon Son Chung, Andrew Senior, Oriol Vinyals and Andrew Zisserman released "Lip Reading Sentences in the Wild", which built the Lip Reading Sentences dataset of over 100,000 natural sentences from BBC programs recorded between 2010 and 2016, and showed the model outperforming a professional lip reader. In 2018, the Oxford group released LRS3, over 400 hours of TED and TEDx talks, which became the main public benchmark. The same year, DeepMind and Google researchers scaled to 3,886 hours from YouTube with a vocabulary of 127,055 words, reaching a 40.9% word error rate where professional lip readers scored 86.4% or 92.9%.

The next leap came from self-supervised learning, which learns from unlabeled data first. Meta's AV-HuBERT (Bowen Shi and colleagues, ICLR 2022) masks parts of audio-visual video and learns to predict clusters of hidden features, then fine-tunes on a small labeled set. With 30 hours of labels it reached 32.5% word error rate on LRS3, beating a previous system trained on about 31,000 hours of transcribed video, and 26.9% with 433 labeled hours plus self-training. In 2023, Auto-AVSR from Imperial College London and Meta used public speech recognizers such as Whisper to automatically transcribe about 2,600 hours of English video from VoxCeleb2 and AVSpeech, training on 3,448 hours in total. Its visual-only model reached 19.1% word error rate on LRS3.

Compared with audio speech recognition, lip reading remains far less accurate. The same Auto-AVSR paper reports audio-visual error of 0.9% on LRS3, roughly 20 times lower than its visual-only result. Video therefore adds the most value as a complement: the 2017 Oxford work showed visual information improves recognition even when audio is available, and AVSR systems hold up better in loud environments. Compared with human lip readers, the best systems are clearly better on benchmark video, but those comparisons use frontal, well-lit footage of the type the models were trained on.

Limitations are significant. The DeepMind authors state plainly that their model only performs well on high-quality video of a subject facing the camera at a certain distance, and not in other contexts. Side angles, low resolution, occlusion by hands or masks, beards, accents, and fast speech all raise error rates. Most large datasets are English and drawn from broadcast or online talks, so performance on other languages and everyday speech is weaker. Unseen speakers are harder: LipNet's word error rate rose from 4.8% to 11.4% on speakers outside training. Building datasets is costly; in DeepMind's YouTube pipeline only about 2% of candidate clips passed the filters.

The technology also raises privacy concerns. The LipNet authors list security and silent-movie processing among potential uses, and a system that reads speech from video could in principle be applied to surveillance footage without a speaker's knowledge, although current accuracy on distant, angled, low-resolution video is poor. On the assistive side, the DeepMind paper notes that 103,925 tracheostomies were performed in the United States in 2014, a procedure that can leave patients unable to voice speech, and frames silent speech recognition as a potential communication aid. Training state-of-the-art models takes substantial compute: the 2017 Oxford model trained for about 10 days, and newer systems use thousands of hours of video.

04Urugero rwakazi

Gukurikirana clip-amasegonda abiri ukoresheje umuyoboro usoma iminwa

  1. Fata videwo y'amasegonda abiri yumuntu uvuga ngo 'shyira umutuku kuri C zeru', yanditswe kuri frame 25 kumasegonda, utange amakaramu 50.

  2. Koresha isura yerekana ibimenyetso kuri buri kintu hanyuma uhinge akarere gafite ibara ryimeza ryerekanwe kumunwa.

  3. Hisha ibihingwa 50 unyuze imbere ya 3D yemeza imbere isohora icyerekezo kimwe kumurongo.

  4. Kugaburira ibice 50 muri Transformer cyangwa kodegisi isubirwamo kugirango buri cyiciro kigaragaza imvugo yose.

  5. Kwemeza hamwe na CTC kumurongo ushakisha wongeyeho icyitegererezo cyururimi, gisenyuka inshuro nyinshi ibisubizo hamwe nibisobanuro mumiterere cyangwa fonema ikurikirana.

  6. Gereranya ibisohoka ninteruro yukuri hanyuma ubare igipimo cyikosa ryijambo nkibisimbuza wongeyeho gusiba wongeyeho ibyongeweho bigabanijwe numubare wamagambo.

Ibyo byerekana

Ibisohoka ni inyandiko ikeka nka 'shyira umutuku kuri C zeru'. Niba icyitegererezo gisubije 'uburiri bwumwanya kuri C zeru', ikosa nimwe risimburwa mumagambo atanu, igipimo cyamagambo 20%, kandi ryerekana p, b na m viseme idasobanutse neza imiterere nururimi bigamije gukemura.

Gushyira mu bikorwa Isi

LipNet yo muri kaminuza ya Oxford na DeepMind (2016) yageze kuri 95.2% byukuri kuri interuro kuri GRID corpus yinteruro ngufi zisa n’amategeko, ugereranije ni 52.3% kubasomyi batatu bafite ubumuga bwo kutumva.

Oxford's Watch, Umva, Kwitabira no Kwandika (CVPR 2017), yatojwe kuri tereviziyo ya BBC, yandukura amagambo agera kuri kimwe cya kabiri mu magambo y’ibizamini, mu gihe umusomyi w’iminwa wabigize umwuga ufite uburambe bwimyaka 10 yabonye munsi ya kimwe cya kane.

DeepMind na Google bubatse dataset yamasaha 3.886 yo gusoma iminwa yo muri YouTube muri 2018 kandi batangaza ko igipimo cyamakosa cyamagambo 40.9%, biterwa nabarwayi babuze ijwi nyuma ya tracheostomy.

AV-HuBERT ya Meta (ICLR 2022) yigiye kuri videwo yerekana amajwi n'amashusho kandi itagera kuri 32.5% igipimo cyo gusoma umunwa kuri LRS3 ukoresheje amasaha 30 gusa yamakuru yanditse.

Kumenyekanisha amajwi-amashusho yerekana guhuza iminwa n'amajwi; sisitemu ya Auto-AVSR (ICASSP 2023) yageze kuri 0.9% igipimo cyamakosa yijambo kurwego rwa LRS3 TED.

Kazoza ka AI mugusoma iminwa no kumenyekanisha imvugo

The clearest trend in published work is that more data, even with imperfect labels, keeps improving visual speech recognition. The Auto-AVSR authors (2023) reported that visual-only error kept falling as automatically transcribed hours were added, while audio-only error saturated beyond about 1,500 hours. That suggests larger, more diverse video collections and better self-supervised pretraining, as in AV-HuBERT (2022), will drive further gains. Open problems include robustness to real-world video with side views, low light and low resolution, which the 2018 DeepMind authors flagged as a hard limit of their system; coverage of languages beyond English; and handling speakers with atypical articulation, who are often the people who would benefit most from assistive lip reading. Benchmarks such as LRS3 consist of TED talks, so results there may overstate performance on everyday conversation. The social questions are as open as the technical ones. Using lip reading on video recorded without consent would raise privacy and civil-liberties issues, and dataset construction from public online video raises consent questions of its own. Whether assistive products for people with voice loss reach routine clinical use depends on accuracy for individual patients rather than benchmark averages.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Kubaka amahitamo. Igishushanyo-cy-urwego rugena niba AI itezimbere ibisubizo nyabyo.
  • Itsinda hamwe nakazi. Guhuza ibikorwa byiza bikora umusaruro wunguka abakoresha bashobora kwizera.
  • Ibyago n'umutekano. Gukoresha neza ibibazo bigabanya umunaniro wimpinduka hamwe ningaruka zo gushyira mubikorwa.

Witondere

  • Gutangiza inzira yamenetse birashobora kongera ibibazo bihari.
  • Amakipe arashobora gukora cyane kandi agakuraho ibitekerezo byabantu bikenewe.
  • Ubwiza burashobora gutemba niba ibisubizo bidahwema gusuzumwa.

Intambwe ikurikira

  1. Shushanya ibikorwa byubu hanyuma umenye intambwe-yo guterana hejuru.
  2. Sobanura aho abantu bagenzura mbere yo kwikora byuzuye.
  3. Hugura abakoresha kubisobanuro, inzira zo kuzamuka, hamwe nubuziranenge.
  4. Kurikirana ibisubizo-urwego rwibisubizo kugirango wemeze agaciro karambye.

Inkomoko no gusoma

  1. arXivLipNet: End-to-End Sentence-level Lipreading
  2. arXiv (CVPR 2017)Amagambo yo Gusoma Iminwa mu gasozi
  3. arXivLRS3-TED: dataset nini nini yo kumenya imvugo
  4. arXivKinini-Igipimo Cyerekanwa Cyimvugo Kumenyekana
  5. arXiv (ICLR 2022)Kwiga Amajwi-Amashusho Yerekana Guhagararirwa na Masked Multimodal Cluster Prediction (AV-HuBERT)
  6. arXiv (ICASSP 2023)Auto-AVSR: Ijwi-Amashusho Imvugo Kumenyekanisha hamwe na Automatic Labels

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Fata AI mu Gusoma Iminwa no Kubona Imvugo Yerekana Ikibazo

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

AI ni iki mu Gusoma Iminwa no Kumenyekanisha Imvugo?

Kumenyekanisha imvugo (VSR), cyangwa gusoma iminwa ya AI, byandukura imvugo ivuye mumashusho yumunwa wumuvugizi udakoresheje amajwi. Kuva mu mwaka wa 2016, sisitemu yo kwiga yimbitse yatojwe kumasaha ibihumbi ya videwo yavuye mumagambo mato mato ajyanye no kwandikirana amagambo, hamwe n’ikosa ry’amagambo hafi 19% ku biganiro bya TED mu 2023. Bifite akamaro ku bantu badashobora kuvuga amajwi, ahantu h’urusaku, no kujya impaka ku buzima bwite.

Ni ubuhe buryo budasobanutse itsinda rya viseme risobanura mu kumenyekanisha imvugo?

Amatsinda nka p, b, na m yerekana impamvu igaragara neza yonyine idashobora gutandukanya foneme.

Nigute umuyoboro wamashusho ushobora gufasha amajwi-amashusho kumenya mugihe urusaku rwa acoustic rwiyongereye?

Uburyo burashobora kuzuzanya, hamwe ninyungu bitewe nubushakashatsi hamwe nicyitegererezo.

Niki imbere ya 3D yemeza imbere irangira muburyo bwo gusoma iminwa ifasha gufata?

3D imyanzuro itunganya ama frame menshi hamwe, ifata uburyo umunwa ugenda mugihe gito aho kuba ishusho imwe ihamye.

Ni ubuhe buryo bwo gufata bushobora guhisha amakuru akenewe muburyo bwo kwerekana-imvugo?

Kugaragara no gufata ubuziranenge bigira ingaruka kumunwa amakuru aboneka kubitekerezo.

Ni ubuhe buryo bwo gusuzuma bugomba guhuzwa na LipNet yumwimerere ibisubizo-kumenyekanisha ibisubizo?

Ibipimo ngenderwaho ntabwo bishyiraho imikorere yisi yose mundimi, abavuga, hamwe nuburyo bwo gufata.