Evaluating AI Video Generators with VBench

VBench was introduced in late 2023 by researchers from Nanyang Technological University's S-Lab and Shanghai AI Laboratory, and presented at CVPR 2024. Its core idea is decomposition. Instead of one score such as FVD (Frechet Video Distance), it measures 16 dimensions grouped into two families. Video quality dimensions judge the clip itself, regardless of the prompt: subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality and imaging quality. Video-condition consistency dimensions judge how well the video follows its prompt: object class, multiple objects, human action, color, spatial relationship, scene, appearance style, temporal style and overall consistency. Each dimension has its own prompt suite and a specialized evaluator, usually an existing vision model. Subject consistency compares DINO features across frames. Background consistency uses CLIP features. Dynamic degree uses optical flow. Aesthetic and imaging quality use learned quality predictors, and object and color checks use detection or captioning models. The authors collected human preference annotations for each dimension and reported that the automatic scores agree well with them. The breakdown exposes trade-offs that a total hides. A nearly frozen video can score very well on consistency and smoothness while scoring poorly on dynamic degree, so reading only the average can reward timid models. The VBench total is not a complete ranking of video models; it reflects the chosen prompts, evaluators and weights. Later work extended the suite. VBench++ added image-to-video and trustworthiness evaluation, and VBench-2.0 targets deeper properties such as physics, commonsense and human anatomy. Public arenas, where people vote between two anonymous clips from the same prompt, capture overall preference, including qualities such as physically plausible motion that automated evaluators do not yet measure reliably.