視覺人工智慧指南

視訊幀插值

影片畫面插值是將兩個現有畫面之間產生新畫面的任務,讓影片播放更順暢或慢動作。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 使用 RIFE 將影片幀率加倍
  5. 現實世界的實施
  6. 視訊幀插值的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

現代方法使用神經網路估算每個像素的移動,然後扭曲並混合相鄰畫面。相同概念驅動慢動作工具、幀率轉換器與遊戲中的畫面產生。

重點摘要

  1. 影片畫面插值在現有畫面之間產生新畫面,通常透過估算運動並將相鄰畫面向目標時間扭曲。
  2. Super SloMo (CVPR 2018) 結合雙向光流與可見度圖,能產生任意數量的中間畫面。
  3. RIFE (ECCV 2022) 直接估算中間光流,作者報告其速度比 Super SloMo 與 DAIN 快 4 至 27 倍。
  4. NVIDIA 的 DLSS 3 畫面產生,於 2022 年 9 月公布,將相同概念即時應用於 RTX 40 系列 GPU 的遊戲。
  5. 插值畫面是預測值,因而在大幅運動、遮擋與薄物件上失效,且不會新增真實資訊。

深入探討

Video frame interpolation (VFI) means synthesizing frames that were never filmed. Given frame A at time 0 and frame B at time 1, the goal is a believable frame at some time t in between, often t = 0.5 for doubling the frame rate. The simplest approaches either repeat a frame or blend A and B, but blending produces ghosted double images whenever something moves. Learned interpolation instead tries to work out where each object would be at time t and draw it there, which is why it can produce smooth slow motion from footage recorded at normal speed.

Most modern methods rely on optical flow, a map that gives every pixel a motion vector showing where it moves between two frames. A typical flow-based pipeline works in four steps. First, estimate flow between A and B in both directions. Second, scale and combine those flows to approximate the flow from the unknown middle frame back to each input. Third, warp, meaning resample, A and B along those flows so both line up with time t. Fourth, fuse the two warped images with weights that say which input to trust at each pixel, because some pixels are hidden in one frame and visible in the other, a problem called occlusion. A refinement network then fixes remaining errors.

NVIDIA's Super SloMo, presented at CVPR 2018, is the clearest example of this design. One U-Net, a common encoder-decoder image network, estimates the bidirectional flow; the flows are linearly combined for each time step; and a second U-Net refines the flows and predicts soft visibility maps that exclude occluded pixels before fusion. Because none of its learned parameters depend on time, it can produce any number of intermediate frames. It was trained on 1,132 clips recorded at 240 frames per second, about 300,000 frames, where real intermediate frames provide the ground truth.

A second family skips explicit flow. Kernel-based methods predict, for every output pixel, a small filter that is applied to patches of the two input frames. Niklaus, Mai and Liu's adaptive separable convolution (ICCV 2017) replaced large 2D kernels with pairs of 1D kernels, cutting the parameters needed to handle large motion and allowing a whole frame to be generated in one pass, which in turn made it practical to train with a perceptual loss. Kernel methods handle some effects that flow struggles with, but their reach is limited by kernel size.

Later work focused on speed and large motion. RIFE, accepted to ECCV 2022, estimates intermediate flow directly with a network called IFNet instead of computing two-way flow and then approximating, and its authors report it is 4 to 27 times faster than Super SloMo and DAIN while scoring better. FILM, from Google Research at ECCV 2022, targets large motion between near-duplicate photos with a multi-scale feature extractor that shares weights across scales, needs no pretrained flow or depth networks, and uses a Gram matrix loss to keep results sharp; it reports leading results on the Xiph large-motion benchmark. In September 2022 NVIDIA brought the idea to games with DLSS 3, which uses current and prior frames, hardware optical flow and engine data such as motion vectors and depth to generate a whole new frame.

The main alternatives are capture and rendering. Filming at a higher frame rate gives true intermediate frames but needs a capable camera and more light. In games, rendering more frames costs GPU time, which is why frame generation is attractive. Simple frame repetition or blending costs almost nothing but either stutters or ghosts on motion. Generative video models can also create in-between frames, and LDMVFI (AAAI 2024) explicitly frames interpolation as conditional generation with a latent diffusion model.

Interpolation fails in predictable ways. Fast or large motion breaks flow estimation and produces smearing or tearing. Occlusions, transparent or reflective surfaces, repeating textures, motion blur, scene cuts and flickering lights confuse the motion model. Thin objects such as fences and spokes often break apart. Interpolated frames also add no new information: a generated frame is a guess, so it should not be treated as evidence in forensic or scientific footage. In games, a generated frame cannot react to the newest input, which is why NVIDIA pairs DLSS 3 with its Reflex latency-reduction technology. Evaluation is also tricky: the LDMVFI authors argue that common pixel metrics such as L1 and L2 distance are poor indicators of how good an interpolated frame looks.

In practice, open-source models run on consumer GPUs. RIFE's repository reports more than 30 fps for 2x interpolation of 720p video on an RTX 2080 Ti and offers simple options such as --exp=1 for 2x and --exp=2 for 4x output. FILM's repository lists CUDA 11.2.1 and cuDNN 8.1.0 as tested versions, can split high-resolution frames into patches, and writes interpolated frames and an MP4. Training a model from scratch requires high-frame-rate video so that real middle frames can be held out as targets. DLSS 3 frame generation, by contrast, is limited by NVIDIA to RTX 40 Series hardware.

04工作範例

使用 RIFE 將影片幀率加倍

  1. 複製 ECCV2022-RIFE 儲存庫,安裝其需求並按照 README 的說明下載預訓練模型檔案。

  2. 挑選一段具有適度運動的短 30 fps 影片片段(例如人走路),並將其儲存為 video.mp4。

  3. 執行 python inference_video.py --exp=1 --video=video.mp4 以產生 60 fps 的 2 倍版本。

  4. 再次執行並加上 --exp=2 以產生 4 倍版本進行比較。

  5. 逐影格檢查輸出結果,並標記生成的影格在何處出現殘影、拉伸或細長物體斷裂的情況。

  6. 使用具有快速平移或場景切換的片段重複上述步驟,以觀察運動模型在何處失效。

它顯示了什麼

學習者將擁有更流暢的 60 fps 和 120 fps 版本片段以及一份失敗案例清單。若要客觀檢查品質,請刪除真實 60 fps 片段中的每隔一個影格,將其插補回來,並使用 PSNR 或 SSIM 將生成的影格與保留的原始影格進行比較。

現實世界的實施

NVIDIA 的 DLSS 3 於 2022 年 9 月針對 GeForce RTX 40 系列顯示卡發布,它利用先前與當前的影格、硬體光流場以及引擎運動向量,在渲染的遊戲影格之間生成一個完整的額外影格。

Google Research 推出的 FILM 於 ECCV 2022 發表,能將兩張幾乎相同的照片轉化為短片的慢動作剪輯,並以 Apache-2.0 授權條款作為開源的 TensorFlow 程式碼釋出。

開源的 RIFE 模型被用於 Flowframes、SVP 與 Autodesk Flame 等工具中,以提升影片的幀率,作者報告在 RTX 2080 Ti 上將 720p 影片加倍可達每秒超過 30 幀。

NVIDIA 的 Super SloMo 於 CVPR 2018 發表,訓練了 1,132 支以每秒 240 幀拍攝的剪輯,能產生任意數量的中間幀,將普通影像轉換為慢動作。

FILM 的命令列工具可將一個快照相簿轉成平滑影片:每次額外的處理會將中間幀數加倍,對於 n 次處理會產生 (2^n + 1) × (frames - 1) 輸出幀,並以影像與 MP4 形式寫出。

視訊幀插值的未來

Two trends are visible in recent research. First, interpolation is converging with video generation. LDMVFI, accepted at AAAI 2024, treats interpolation as conditional generation with a latent diffusion model and reports better perceptual quality than earlier methods in quantitative tests and a user study. Second, real-time frame generation has moved into consumer products: NVIDIA's DLSS 3, announced in September 2022, generates whole frames in games on dedicated hardware. Open problems remain. Very large motion, occlusion and thin structures still cause visible errors, which is why FILM's 2022 paper focused on large-motion benchmarks. Evaluation is unsettled, since pixel metrics such as PSNR and L1 can disagree with human judgments of sharpness and plausibility, as the LDMVFI authors argue. Latency is a constraint for interactive use, because generating a frame between two rendered ones means waiting for the second. There are also questions of trust. As generated frames become harder to spot, footage used for journalism, science or legal evidence may need clear labeling of which frames were captured and which were synthesized. How such provenance will be recorded is still being worked out across the industry.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXiv (CVPR 2018)Super SloMo:高品質多中間幀估計用於影片插值
  2. arXiv (ICCV 2017)透過自適應可分離卷積的影片幀插值
  3. arXiv (ECCV 2022)即時中間流估計用於影片幀插值
  4. GitHubECCV2022-RIFE 倉庫
  5. arXiv (ECCV 2022)FILM:大幅位移的幀插值
  6. Google Research on GitHubFILM:大幅位移的幀插值(官方程式碼)
  7. NVIDIANVIDIA DLSS 3:AI 驅動的效能倍增器可將幀率提升至最高 4 倍
  8. arXiv (AAAI 2024)LDMVFI:使用潛在擴散模型的影片幀插值

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加影片幀插值測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是視訊幀插值?

影片幀插值是指在影片的兩個既有幀之間產生新幀的任務,使畫面播放更流暢或呈現慢動作。現代方法使用神經網路估計每個像素的移動,然後對相鄰幀進行扭曲與混合。相同的概念驅動慢動作工具、幀率轉換器以及遊戲中的幀生成。

視訊幀插值為影片剪輯添加了什麼?

幀插值在現有幀之間合成合理的中間幀,平滑運動或實現慢動作播放。

為什麼插值不能簡單地平均(混合)兩個相鄰幀?

由於物件在幀之間移動,簡單的混合會產生重影;插值器必須考慮運動,以便在中間時間正確放置物件。

幀插值中的光流是什麼?

光流估計幀之間每個像素的運動,允許模型將內容扭曲到正確的中間位置。

為什麼像 DAIN 這樣的模型要包含深度資訊?

深度感知使模型能够解决变形过程中的遮挡问题,因此较近的物体可以正确地与远处的物体重叠,从而减少伪影。

「去遮擋」是指插值中的挑戰是什麼?

隨著物件移動,先前隱藏的背景變得可見並且必須進行合成,因為它沒有完全存在於兩個輸入幀中。