概述
由於運動模組經過一次訓練並插入到基於同一基礎構建的任何個性化模型中,因此無需重新訓練即可對社區藝術風格進行動畫處理。
重點摘要
- AnimateDiff 將單獨訓練的運動模組添加到凍結的 Stable Diffusion 模型中,因此其個人化變體可以產生短動畫而無需重新訓練。
- 運動模組是一個時間 Transformer,它在每個空間位置的幀之間運行自註意力,並使用正弦編碼來標記幀順序。
- 訓練使用三個階段:視訊框架上的網域適配器、WebVid-10M 剪輯上的運動模組以及用於特定攝影機移動的可選 MotionLoRA。
- 用於新相機運動的 MotionLoRA 可以在大約一到兩個小時內訓練 20 到 50 個參考視頻,並儲存在大約 30 MB 中。
- 預設的 AnimateDiff 剪輯為 16 幀,每秒 8 幀大約兩秒,較長的影片需要 FreeNoise 等擴展。
深入探討
AnimateDiff answers a narrow but practical question: how do you make an image model move? By 2023, a large community had trained and shared many personalized text-to-image (T2I) models, fine-tuned versions of Stable Diffusion that draw a specific anime style, a photorealistic look or a single character, and shared them on platforms such as Civitai and Hugging Face. Those models only produced still images. Training a full video model for each one would be far too expensive for hobbyists. AnimateDiff, described in "AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning" by Yuwei Guo and colleagues (arXiv July 2023, ICLR 2024), adds a reusable motion module that supplies movement while the original model keeps supplying appearance.
Some background helps. Stable Diffusion is a latent diffusion model: an autoencoder compresses images into a smaller latent grid, and a U-Net, a convolutional network with attention layers, learns to remove noise from those latents step by step, guided by a text prompt. A personalized model changes the U-Net's weights, often through LoRA (low-rank adaptation), which adds small trainable matrices to attention layers. AnimateDiff's trick is to leave every one of those image layers untouched and add new layers between them that only look across time.
The method has three training stages. First, a domain adapter, a small set of LoRA layers, is fine-tuned on individual frames from the WebVid-10M video dataset. The paper notes that video frames are visually poorer than curated image datasets, so the adapter absorbs that quality gap and is switched off at inference time. Second, the image model is inflated so that it treats a video as a batch of separate frames, and a new motion module is inserted after the existing blocks at each resolution. Only the motion module is trained on video clips; the base model and adapter stay frozen. Third, optional MotionLoRA layers are trained on the motion module to learn a specific camera move from as few as 20 to 50 reference videos in about 2,000 iterations, roughly one to two hours, producing a file of about 30 MB. At generation time the user loads any personalized model built on the same base, adds the motion module and any MotionLoRA, and samples all frames together.
The motion module itself is a temporal Transformer. At each U-Net location, it treats the sequence of feature vectors across frames as a short sequence, adds sinusoidal position encodings so the model knows frame order, and runs self-attention along the time axis, so each frame's features can borrow information from every other frame. The authors report that the position encoding is essential; without it the module cannot tell the frames apart. The output projection is initialized to zero with a residual connection, so at the start of training the module does nothing and cannot damage the image model. The paper also compared this design against one-dimensional temporal convolutions with a similar parameter count and found attention adequate for learning motion.
Releases followed quickly. According to the official GitHub repository, version 1 motion modules for Stable Diffusion 1.4 and 1.5 appeared in July 2023 with 417 million parameters each. Version 2 in September 2023 was trained at larger resolution and batch size and came with eight camera-motion MotionLoRAs. A beta motion module for SDXL followed in November 2023, trained at 1,024 by 1,024 with 16 frames and needing about 13 GB of GPU memory. Version 3 and SparseCtrl encoders for sketches and RGB images arrived in December 2023. The code is released under the Apache 2.0 license and described by the authors as for academic use. Hugging Face's Diffusers library added AnimateDiffPipeline, video-to-video, ControlNet and SDXL variants.
Compared with alternatives, AnimateDiff trades scale for flexibility. Large video models such as Stable Video Diffusion (Stability AI, November 2023) are trained end to end in stages on curated video, giving stronger motion but fixing a single visual style unless fine-tuned. Tune-A-Video fine-tunes a model on one input video, and Text2Video-Zero animates an image model without training by warping latents with predefined motion, both cited by the AnimateDiff paper as related work. AnimateDiff's advantage is that one motion module works across many community models, preserving their look. In the paper's user study and CLIP-based measurements, it was compared against Text2Video-Zero and Tune-A-Video on text alignment, domain similarity to the personalized model, and motion smoothness, and it was also shown alongside the commercial tools Runway Gen-2 and Pika Labs.
The limits are real. Clips are short: the Diffusers default is 16 frames, about two seconds at 8 frames per second, and longer videos need add-ons such as FreeNoise that stitch overlapping windows. Motion is learned from WebVid's mostly real-world recordings, so the range of movements the module knows reflects that data, and frames can drift or flicker; FreeInit, an inference-time add-on in Diffusers, exists specifically to improve temporal consistency at extra compute cost. A motion module only works with personalized models from the same base, so a Stable Diffusion 1.5 module does not plug into SDXL models. The Diffusers documentation warns that checkpoints are sensitive to the noise schedule settings, recommending a linear beta schedule and disabling sample clipping. Output quality depends heavily on which personalized model is used.
In practice AnimateDiff runs on a single consumer GPU for Stable Diffusion 1.5 models, using techniques such as CPU offloading and sliced VAE decoding that Diffusers exposes, while the SDXL beta needs about 13 GB of video memory. Training a new motion module from scratch requires a video dataset and substantial GPU time, but training a MotionLoRA for a new camera move is within reach of individuals, at around an hour or two on the authors' reported setup.
04工作範例
使用擴散器產生 16 幀剪輯
安裝擴散器、變壓器和手電筒,如果記憶體有限,請使用啟用了 CPU 卸載的 GPU。
使用 MotionAdapter.from_pretrained 載入運動適配器 guoyww/animatediff-motion-adapter-v1-5-2。
將 Stable Diffusion 1.5 微調(例如 SG161222/Realistic_Vision_V5.1_noVAE)載入到 AnimateDiffPipeline 中,並將 Motion_adapter 設定為此轉接器。
依照 Diffusers 文件的建議,設定 DDIM 調度程序,其中 Clip_sample 設定為 False,beta_schedule 設定為 Linear。
使用“日落、橙色天空、海浪、海鷗”等提示、num_frames=16、guiding_scale=7.5、25 個推理步驟和固定種子生成。
將幀匯出為 GIF,然後使用 load_lora_weights 載入縮小的 MotionLoRA,並使用相同的種子再次產生。
它顯示了什麼
兩個大約兩秒的短片共享相同的場景和風格,一個具有模組的預設運動,另一個具有穩定的縮小效果。透過逐步瀏覽幀並觀察波峰等細節中的閃爍來檢查時間一致性。
現實世界的實施
藝術家將 AnimateDiff 運動模組與 Stable Diffusion 1.5 的社群微調(例如 Hugging Face Diffusers 文件中使用的現實視覺模型)相結合,以該模型的風格製作短循環剪輯。
與 AnimateDiff v2 一起發布的 MotionLoRA 檢查點添加了八種相機移動,包括放大、縮小、向左平移、向右平移、向上傾斜、向下傾斜以及順時針或逆時針滾動。
SparseCtrl 由同一團隊於 2023 年 12 月發布,可讓用戶使用一個或幾個草圖或 RGB 關鍵影格來引導 AnimateDiff 剪輯,這對於故事板和關鍵影格動畫非常有用。
Diffusers 庫的 FreeNoise 選項透過隨時間滑動視窗並更改所選幀的提示,將 16 幀 AnimateDiff 模型擴展到數百幀的剪輯。
AnimateLCM 將 AnimateDiff 式運動模組與一致性 LoRA 配對,因此可以透過大約六個降噪步驟(而不是數十個)對 16 幀剪輯進行取樣。
AnimateDiff 運動生成的未來
AnimateDiff's plug-in approach sparked a wave of add-ons in late 2023, including SparseCtrl for keyframe control, AnimateLCM for few-step sampling, and FreeNoise and FreeInit for longer and more consistent clips, all now exposed in the Diffusers library. The open questions it leaves are the general ones in video generation: longer duration without drift, physically plausible motion, and consistent identity of characters and objects across frames. The broader field has moved toward larger video models trained end to end on curated data, as in Stable Video Diffusion (November 2023), and toward transformer backbones rather than image U-Nets. Whether lightweight motion adapters remain useful as base models change is unsettled: a motion module is tied to the base model it was trained with, so each new image architecture needs its own. The AnimateDiff authors' own SDXL module was released as a beta, illustrating that porting the idea to newer bases is not automatic. There are also policy questions. Personalized models can imitate specific artists or real people, and animating them makes synthetic media more convincing. The project's statement that it is released for academic use and the licenses of the community models it animates both shape how it can be used, and detection and provenance tools for short AI-generated clips are still developing.
付諸實踐
為什麼這很重要
- 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
- 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
- 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。
留意
- 如果出處不明,肖像權和同意可能會成為法律風險。
- 模型表現可能因光照、人口統計和環境的不同而有所不同。
- 除非監控置信閾值,否則誤報可能會被忽略。
後續步驟
- 定義精確度、召回率和錯誤成本的接受標準。
- 使用符合實際生產條件的數據進行測試。
- 為低置信度或高影響力的預測添加人工審核。
- 追蹤模型漂移並在相機或資料集變更後重新驗證。
資料來源與延伸閱讀
- arXiv (ICLR 2024)AnimateDiff:無需特定調整即可為您的個人化文字到圖像擴散模型製作動畫
- GitHubAnimateDiff 官方儲存庫
- Hugging Face Diffusers documentationAnimateDiff 管道
- arXivSparseCtrl:為文字到視訊擴散模型添加稀疏控件
- arXivStable Video Diffusion:將潛在視頻擴散模型擴展到大型數據集
- arXivLoRA:大型語言模型的低秩適配。
不斷探索
常見問題
什麼是 AnimateDiff 運動生成?
AnimateDiff 是由香港中文大學、上海人工智慧實驗室和史丹佛大學的研究人員在 ICLR 2024 上發表的一種方法,透過插入單獨訓練的運動模組,將靜態圖像 Stable Diffusion 模型轉變為短視頻生成器。由於運動模組經過一次訓練並插入到基於同一基礎構建的任何個性化模型中,因此無需重新訓練即可對社區藝術風格進行動畫處理。
AnimateDiff 背後的核心思想是什麼?
AnimateDiff 將單獨訓練的運動模組插入現有的凍結文字到圖像模型中,這樣它就可以產生運動,而無需重新訓練基礎模型。
重複使用 AnimateDiff 運動模組時,哪些檢查點相容性條件很重要?
使用固定空間權重進行訓練可以在支援的模型系列中進行重複使用,而不是通用插件相容性。
運動模組如何保持幀之間的一致性?
新增至 U-Net 的時間注意力層讓每個幀沿著時間軸專注於其他幀,從而保持一致性。
AnimateDiff 與擴展最相關的是哪個模型系列?
AnimateDiff 旨在為穩定擴散樣式的文字到影像擴散模型添加運動。
AnimateDiff 生態系中的運動 LoRA 通常會控制什麼?
引入運動 LoRA 來控制攝影機運動,例如平移、縮放和滾動。