視覺人工智慧指南

穩定的視訊擴散

Stable Video Diffusion (SVD) 是 Stability AI 於 2023 年 11 月發布的開放權重潛在影片擴散模型,它能將單張靜態圖像轉換為 14 或 25 幀、解析度為 576x1024 的短片。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 使用 SVD-XT 從一張圖像生成短片
  5. 現實世界的實施
  6. 穩定影片傳播的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

這之所以重要,是因為其論文和公開權重詳細展示了數據策劃和分階段訓練如何將圖像生成器轉變為影片生成器,並成為攝影機控制和 3D 研究的基礎。

重點摘要

  1. Stable Video Diffusion 是 Stability AI 於 2023 年 11 月 21 日發布的開放權重圖像轉影片模型,提供 14 幀和 25 幀版本,解析度為 576x1024。
  2. 它在 Stable Diffusion 2.1 圖像模型的基礎上增加了時間卷積層和注意力層,並對整個網路進行了影片微調。
  3. 其論文使用了一組經過篩選的 1.52 億個剪輯片段進行訓練,這些片段是從約 5.8 億個片段中,透過鏡頭切換檢測、光流、OCR 和 CLIP 分數篩選而來。
  4. 模型卡片列出了大約四秒以內的短片、無文字控制,以及在臉部、人物和清晰文字上的弱點。
  5. 目前的 Stability AI 社群授權(Community License)僅允許年營收低於 100 萬美元的組織進行商業使用。

深入探討

Stable Video Diffusion is a family of video generation models from Stability AI, the company behind Stable Diffusion. The publicly released checkpoints are image-to-video models: given one still picture, they generate a short video that starts from that picture and adds motion. The original SVD checkpoint produces 14 frames, and SVD-XT, finetuned from it, produces 25 frames at 576x1024 pixels. Stability AI announced the models on November 21, 2023 as a research preview, saying they were not yet intended for real-world or commercial applications, and published code on GitHub, weights on Hugging Face, and a paper on arXiv.

To understand how it works, start with diffusion. A diffusion model is trained by taking real data, adding random noise in many small steps, and teaching a neural network to predict and remove that noise. To generate something new, the model starts from pure noise and removes it step by step until a clean sample remains. A latent diffusion model does this not on raw pixels but in a compressed space produced by an autoencoder, a network that squeezes an image into a smaller grid of numbers, called latents, and can expand it back. Working in this smaller space is what made Stable Diffusion practical on ordinary GPUs.

SVD extends that idea from images to video. The paper, by Andreas Blattmann and colleagues, starts from the Stable Diffusion 2.1 image model and inserts temporal layers, a temporal convolution and a temporal attention layer after every spatial layer, so the network can relate each frame to the others. The whole network is then trained on video. In the image-to-video version, the input picture is used as conditioning: it guides every denoising step so the clip stays consistent with it. Extra inputs, which the paper calls micro-conditioning, tell the model the target frame rate and, in the released pipeline, how much motion to produce. A decoder finetuned for temporal consistency turns the final latents back into frames.

The paper's main contribution is a recipe in three stages. Stage one is ordinary text-to-image pretraining, inherited from Stable Diffusion. Stage two is video pretraining on a large, filtered dataset at low resolution: 14 frames at 256x384 for 150,000 iterations. Stage three is finetuning on a smaller, high-quality video set at higher resolution. The authors built a Large Video Dataset of about 580 million annotated clips, roughly 212 years of footage, then filtered it to 152 million clips. Filtering removed scene cuts with a cut detector, nearly static clips using optical flow, clips heavy with on-screen text using optical character recognition, and low-quality clips using CLIP-based aesthetic and text-image similarity scores. Captions came from an image captioner, a video captioner and a language model that merged the two. The authors chose the filtering thresholds using human preference ratings and argue that a well-curated pretraining set is essential for high-quality video.

On results, the paper reports a zero-shot Fréchet Video Distance of 242.02 on the UCF-101 benchmark, lower and therefore better than 355.20 for PYOCO and 367.23 for Make-A-Video. The Hugging Face model card states that human voters preferred SVD image-to-video over the commercial tools Runway GEN-2 and PikaLabs for video quality at the time. The paper also showed SVD as a general motion prior: it could be adapted with camera-motion LoRA modules, small add-on weight updates that steer motion, and finetuned into a multi-view model that generates several views of an object at once. Stability AI later built SV3D, published in March 2024, on the same idea, generating orbital videos around an object from a single image for 3D reconstruction.

Compared with alternatives, SVD sits between closed commercial video generators and later open models. Closed services such as Runway's GEN-2 offered text prompts and web interfaces, while SVD offered downloadable weights that researchers could inspect, finetune and run locally. Its main limits relative to newer systems are length and control. The model card lists clips of up to about four seconds, outputs that may show little motion or only slow camera pans, no control through text in the released image-to-video checkpoints, faces and people that may not render properly, and no legible text. The autoencoder is lossy, so fine detail can be lost.

Licensing needs attention. The SVD-XT repository now carries the Stability AI Community License, last updated July 5, 2024. It allows commercial use after registering with Stability AI, but the license terminates for organizations whose annual revenue exceeds US$1 million, which then need a separate license. It also requires a "Powered by Stability AI" notice and bars using the model or its outputs to build or improve other foundation generative models. The model card lists generating factual representations of real people or events as out of scope.

In practice, SVD runs in the diffusers library with a few lines of Python. The diffusers guide shows SVD-XT loaded in 16-bit precision and says that model offloading, feed-forward chunking and decoding one or two frames at a time can bring memory use below 8 GB of VRAM, at the cost of speed and possibly some flicker. Training is another matter: the model card reports about 200,000 A100 80 GB GPU-hours for SVD, roughly 64,000 kWh of energy and about 19,000 kg of CO2 equivalent.

04工作範例

使用 SVD-XT 從一張圖像生成短片

  1. 在配備 NVIDIA GPU 的機器上安裝 diffusers、transformers 和 accelerate Python 套件。

  2. 以 float16 格式從 stabilityai/stable-video-diffusion-img2vid-xt 載入 StableVideoDiffusionPipeline,並呼叫 enable_model_cpu_offload()。

  3. 載入一張風景照片並將其調整為剛好 1024x576 像素,以便與模型的訓練解析度相符。

  4. 固定亂數種子(random seed),然後以 decode_chunk_size=8 生成影格,並以每秒 7 個影格的速度將其匯出為 MP4。

  5. 使用相同的種子再次生成,但如 diffusers 指南所述,設定 motion_bucket_id=180 且 noise_aug_strength=0.1。

  6. 如果記憶體耗盡,請啟用前饋分塊(feed-forward chunking)並將 decode_chunk_size 降低至 2。

它顯示了什麼

兩個各 25 個影格的剪輯片段,在 7 fps 下約為 3.6 秒,兩者皆從同一張圖片開始。檢查第一格是否與輸入高度吻合,並比較這兩個片段:第二個片段應該展示更多動態,且比原圖偏移更多。

現實世界的實施

設計師透過 Hugging Face diffusers 庫將 1024x576 的產品照片輸入至 SVD-XT 檢查點,並獲得一個長達約四秒、展示柔和動態或緩慢鏡頭平移的 25 格剪輯片段。

Stability AI 的 SV3D 在 2024 年 3 月的 arXiv 論文中被描述,它將 SVD 影像轉影片模型調整為從單一相片生成圍繞物件的軌道影片,然後使用這些視角來建構 3D 模型。

SVD 論文的作者將基礎模型微調為 SVD-MV(一個多視角生成器),在 Google Scanned Objects 資料集的 50 個物件上獲得了 0.14 的 LPIPS 分數,領先 SyncDreamer 和 Zero123XL。

研究人員將小型相機動態 LoRA 模組附加到 SVD 基礎模型上,以引導平移或縮放等動態,這是 2023 年 11 月論文中展示的一項調整。

擁有消費級 GPU 的業餘愛好者可以使用模型 CPU 卸載、前饋分塊和較小的解碼分塊大小來執行 SVD-XT,diffusers 指南指出這可以將記憶體需求降至 8 GB VRAM 以下。

穩定影片傳播的未來

SVD's influence now lies mostly in its recipe and its role as a base model rather than in its raw output quality. The November 2023 paper argued that careful data curation and a staged training schedule matter as much as architecture, and that a video model learns a general sense of motion and 3D structure that other tasks can reuse. Stability AI's own March 2024 SV3D paper followed that path, using the image-to-video model with explicit camera control to generate consistent orbital views for 3D reconstruction. Open problems visible in the model card remain active research areas across video generation: longer clips beyond a few seconds, reliable motion when the scene should move, text control in image-to-video models, correct rendering of people and faces, and legible text in frames. Evaluation is also unsettled. Metrics such as Fréchet Video Distance on UCF-101, used in the SVD paper, capture only part of what viewers care about, so human preference studies remain common. Licensing and provenance shape adoption. The Stability AI Community License, updated in July 2024, permits commercial use only below a US$1 million annual revenue threshold, and its acceptable use policy excludes some applications. Teams considering SVD should check the current license file and model card before building on it, since terms can change.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXiv (Blattmann et al., 2023)Stable Video Diffusion:將潛在視頻擴散模型擴展到大型數據集
  2. Stability AI推出 Stable Video Diffusion
  3. Hugging Face (Stability AI)stabilityai/stable-video-diffusion-img2vid-xt 模型卡
  4. Hugging Face (Stability AI)Stability AI 社群授權協議 (SVD-XT 存儲庫)
  5. Hugging Face diffusers documentationStable Video Diffusion 指南
  6. arXiv (Voleti et al., 2024)SV3D:利用潛在視頻擴散從單張圖像創建新穎的多視角合成與 3D 生成

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參與 Stable Video Diffusion 測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是穩定影像擴散?

Stable Video Diffusion (SVD) 是 Stability AI 於 2023 年 11 月發布的開放權重潛在影片擴散模型,能將單張靜態圖片轉化為解析度 576x1024、長度為 14 或 25 格的短片。它之所以重要,是因為其論文與公開權重詳細展示了資料策展與分階段訓練如何將影像生成器轉變為影片生成器,並成為相機控制與 3D 研究的基礎。

公開的 Stable Video Diffusion 影像轉影片檢查點使用什麼輸入?

公開的影像轉影片檢查點以靜態圖片作為剪輯生成條件;該論文也研究了文字轉影片的基礎模型。

SVD 在靜態影像穩定擴散架構中加入了什麼來處理運動?

SVD 插入時間注意力和 3D 卷積層,使幀在時間上保持一致。

穩定視訊擴散在什麼樣的空間中執行去雜訊以節省計算?

與穩定擴散一樣,SVD 是一種潛在擴散模型,它以壓縮的潛在表示形式工作,大大減少了計算量。

SVD 的研究發布有何重大意義?

該論文報告稱將發布程式碼與權重,但檢查點授權條款設定了使用條件;公開可用並不代表擁有不受限制的商業權利。

SVD 的公共檢查點通常會產生大約多少幀?

發布的 SVD 檢查點會產生大約 14 到 25 幀的短片。