稳定的视频扩散

Stable Video Diffusion is a family of video generation models from Stability AI, the company behind Stable Diffusion. The publicly released checkpoints are image-to-video models: given one still picture, they generate a short video that starts from that picture and adds motion. The original SVD checkpoint produces 14 frames, and SVD-XT, finetuned from it, produces 25 frames at 576x1024 pixels. Stability AI announced the models on November 21, 2023 as a research preview, saying they were not yet intended for real-world or commercial applications, and published code on GitHub, weights on Hugging Face, and a paper on arXiv.

To understand how it works, start with diffusion. A diffusion model is trained by taking real data, adding random noise in many small steps, and teaching a neural network to predict and remove that noise. To generate something new, the model starts from pure noise and removes it step by step until a clean sample remains. A latent diffusion model does this not on raw pixels but in a compressed space produced by an autoencoder, a network that squeezes an image into a smaller grid of numbers, called latents, and can expand it back. Working in this smaller space is what made Stable Diffusion practical on ordinary GPUs.

SVD extends that idea from images to video. The paper, by Andreas Blattmann and colleagues, starts from the Stable Diffusion 2.1 image model and inserts temporal layers, a temporal convolution and a temporal attention layer after every spatial layer, so the network can relate each frame to the others. The whole network is then trained on video. In the image-to-video version, the input picture is used as conditioning: it guides every denoising step so the clip stays consistent with it. Extra inputs, which the paper calls micro-conditioning, tell the model the target frame rate and, in the released pipeline, how much motion to produce. A decoder finetuned for temporal consistency turns the final latents back into frames.

The paper's main contribution is a recipe in three stages. Stage one is ordinary text-to-image pretraining, inherited from Stable Diffusion. Stage two is video pretraining on a large, filtered dataset at low resolution: 14 frames at 256x384 for 150,000 iterations. Stage three is finetuning on a smaller, high-quality video set at higher resolution. The authors built a Large Video Dataset of about 580 million annotated clips, roughly 212 years of footage, then filtered it to 152 million clips. Filtering removed scene cuts with a cut detector, nearly static clips using optical flow, clips heavy with on-screen text using optical character recognition, and low-quality clips using CLIP-based aesthetic and text-image similarity scores. Captions came from an image captioner, a video captioner and a language model that merged the two. The authors chose the filtering thresholds using human preference ratings and argue that a well-curated pretraining set is essential for high-quality video.

On results, the paper reports a zero-shot Fréchet Video Distance of 242.02 on the UCF-101 benchmark, lower and therefore better than 355.20 for PYOCO and 367.23 for Make-A-Video. The Hugging Face model card states that human voters preferred SVD image-to-video over the commercial tools Runway GEN-2 and PikaLabs for video quality at the time. The paper also showed SVD as a general motion prior: it could be adapted with camera-motion LoRA modules, small add-on weight updates that steer motion, and finetuned into a multi-view model that generates several views of an object at once. Stability AI later built SV3D, published in March 2024, on the same idea, generating orbital videos around an object from a single image for 3D reconstruction.

Compared with alternatives, SVD sits between closed commercial video generators and later open models. Closed services such as Runway's GEN-2 offered text prompts and web interfaces, while SVD offered downloadable weights that researchers could inspect, finetune and run locally. Its main limits relative to newer systems are length and control. The model card lists clips of up to about four seconds, outputs that may show little motion or only slow camera pans, no control through text in the released image-to-video checkpoints, faces and people that may not render properly, and no legible text. The autoencoder is lossy, so fine detail can be lost.

Licensing needs attention. The SVD-XT repository now carries the Stability AI Community License, last updated July 5, 2024. It allows commercial use after registering with Stability AI, but the license terminates for organizations whose annual revenue exceeds US$1 million, which then need a separate license. It also requires a "Powered by Stability AI" notice and bars using the model or its outputs to build or improve other foundation generative models. The model card lists generating factual representations of real people or events as out of scope.

In practice, SVD runs in the diffusers library with a few lines of Python. The diffusers guide shows SVD-XT loaded in 16-bit precision and says that model offloading, feed-forward chunking and decoding one or two frames at a time can bring memory use below 8 GB of VRAM, at the cost of speed and possibly some flicker. Training is another matter: the model card reports about 200,000 A100 80 GB GPU-hours for SVD, roughly 64,000 kWh of energy and about 19,000 kg of CO2 equivalent.