Audio-Driven AI Lip Sync

Wav2Lip, published in 2020 by researchers at IIIT Hyderabad (ACM Multimedia), is the best-known open lip sync model and a useful template for the field. For each target frame, the generator receives three inputs: the face with its lower half masked out, a reference frame of the same person that supplies appearance, and a short mel-spectrogram window of the audio around that moment. An encoder-decoder network then paints in the mouth region, and the result is blended back into the original video. The key contribution was a pre-trained "expert" sync discriminator based on SyncNet. It stays frozen during training and judges whether a short run of mouth frames matches the audio. Earlier approaches trained a sync discriminator jointly with the generator, and those discriminators turned out to be poor judges of sync. Wav2Lip also used a separate visual-quality discriminator. Its weaknesses were low-resolution mouth crops, blurry teeth and visible seams or flicker at the mask boundary. Later systems such as VideoReTalking, MuseTalk and diffusion-based LatentSync aim for sharper, more stable results. Quality is judged in several ways. LSE-D measures the distance between SyncNet's audio and video embeddings, where lower is better. LSE-C measures SyncNet's confidence, where higher is better. Image-quality metrics such as FID and human ratings cover realism. A common misconception is that a good sync score means a realistic video. Exaggerated mouth motion can score well and still look wrong, so human evaluation remains essential. One challenge is that many phonemes share the same visible mouth shape, or viseme, so the audio-to-mouth mapping is many-to-one. Misuse follows directly from the technology: pairing a cloned voice with lip sync can produce convincing fake statements. Researchers have shown that mismatches between phonemes and visemes, especially lips failing to close on 'm', 'b' and 'p', can expose such fakes.