GUIDA AI visiva

Interpolazione dei fotogrammi video

L'interpolazione dei fotogrammi video è il compito di creare nuovi fotogrammi tra due fotogrammi esistenti di un video, in modo che il metraggio possa essere riprodotto in modo più fluido o al rallentatore.

Ascolta questa guidaNarrato dall'IA · ~8 min

In questa pagina9 minuti di lettura
  1. Panoramica
  2. Punti chiave
  3. Immersione profonda
  4. Doubling a clip's frame rate with RIFE
  5. Implementazione nel mondo reale
  6. Il futuro dell'interpolazione dei fotogrammi video
  7. Metterlo in pratica
  8. Fonti e approfondimenti
  9. Continua a esplorare
  10. Domande frequenti

Panoramica

Modern methods use neural networks that estimate how each pixel moves and then warp and blend the neighboring frames. The same idea powers slow-motion tools, frame-rate converters and frame generation in games.

Punti chiave

  1. Video frame interpolation creates new frames between existing ones, usually by estimating motion and warping the neighboring frames toward the target time.
  2. Super SloMo (CVPR 2018) combines bidirectional optical flow with visibility maps and can generate any number of intermediate frames.
  3. RIFE (ECCV 2022) estimates intermediate flow directly and its authors report it is 4 to 27 times faster than Super SloMo and DAIN.
  4. NVIDIA's DLSS 3 frame generation, announced in September 2022, applies the same idea in real time to games on RTX 40 Series GPUs.
  5. Interpolated frames are predictions, so they fail on large motion, occlusions and thin objects and add no new real information.

Immersione profonda

Video frame interpolation (VFI) means synthesizing frames that were never filmed. Given frame A at time 0 and frame B at time 1, the goal is a believable frame at some time t in between, often t = 0.5 for doubling the frame rate. The simplest approaches either repeat a frame or blend A and B, but blending produces ghosted double images whenever something moves. Learned interpolation instead tries to work out where each object would be at time t and draw it there, which is why it can produce smooth slow motion from footage recorded at normal speed.

Most modern methods rely on optical flow, a map that gives every pixel a motion vector showing where it moves between two frames. A typical flow-based pipeline works in four steps. First, estimate flow between A and B in both directions. Second, scale and combine those flows to approximate the flow from the unknown middle frame back to each input. Third, warp, meaning resample, A and B along those flows so both line up with time t. Fourth, fuse the two warped images with weights that say which input to trust at each pixel, because some pixels are hidden in one frame and visible in the other, a problem called occlusion. A refinement network then fixes remaining errors.

NVIDIA's Super SloMo, presented at CVPR 2018, is the clearest example of this design. One U-Net, a common encoder-decoder image network, estimates the bidirectional flow; the flows are linearly combined for each time step; and a second U-Net refines the flows and predicts soft visibility maps that exclude occluded pixels before fusion. Because none of its learned parameters depend on time, it can produce any number of intermediate frames. It was trained on 1,132 clips recorded at 240 frames per second, about 300,000 frames, where real intermediate frames provide the ground truth.

A second family skips explicit flow. Kernel-based methods predict, for every output pixel, a small filter that is applied to patches of the two input frames. Niklaus, Mai and Liu's adaptive separable convolution (ICCV 2017) replaced large 2D kernels with pairs of 1D kernels, cutting the parameters needed to handle large motion and allowing a whole frame to be generated in one pass, which in turn made it practical to train with a perceptual loss. Kernel methods handle some effects that flow struggles with, but their reach is limited by kernel size.

Later work focused on speed and large motion. RIFE, accepted to ECCV 2022, estimates intermediate flow directly with a network called IFNet instead of computing two-way flow and then approximating, and its authors report it is 4 to 27 times faster than Super SloMo and DAIN while scoring better. FILM, from Google Research at ECCV 2022, targets large motion between near-duplicate photos with a multi-scale feature extractor that shares weights across scales, needs no pretrained flow or depth networks, and uses a Gram matrix loss to keep results sharp; it reports leading results on the Xiph large-motion benchmark. In September 2022 NVIDIA brought the idea to games with DLSS 3, which uses current and prior frames, hardware optical flow and engine data such as motion vectors and depth to generate a whole new frame.

The main alternatives are capture and rendering. Filming at a higher frame rate gives true intermediate frames but needs a capable camera and more light. In games, rendering more frames costs GPU time, which is why frame generation is attractive. Simple frame repetition or blending costs almost nothing but either stutters or ghosts on motion. Generative video models can also create in-between frames, and LDMVFI (AAAI 2024) explicitly frames interpolation as conditional generation with a latent diffusion model.

Interpolation fails in predictable ways. Fast or large motion breaks flow estimation and produces smearing or tearing. Occlusions, transparent or reflective surfaces, repeating textures, motion blur, scene cuts and flickering lights confuse the motion model. Thin objects such as fences and spokes often break apart. Interpolated frames also add no new information: a generated frame is a guess, so it should not be treated as evidence in forensic or scientific footage. In games, a generated frame cannot react to the newest input, which is why NVIDIA pairs DLSS 3 with its Reflex latency-reduction technology. Evaluation is also tricky: the LDMVFI authors argue that common pixel metrics such as L1 and L2 distance are poor indicators of how good an interpolated frame looks.

In practice, open-source models run on consumer GPUs. RIFE's repository reports more than 30 fps for 2x interpolation of 720p video on an RTX 2080 Ti and offers simple options such as --exp=1 for 2x and --exp=2 for 4x output. FILM's repository lists CUDA 11.2.1 and cuDNN 8.1.0 as tested versions, can split high-resolution frames into patches, and writes interpolated frames and an MP4. Training a model from scratch requires high-frame-rate video so that real middle frames can be held out as targets. DLSS 3 frame generation, by contrast, is limited by NVIDIA to RTX 40 Series hardware.

04Esempio lavorato

Doubling a clip's frame rate with RIFE

  1. Clone the ECCV2022-RIFE repository, install its requirements and download the pretrained model files as its README describes.

  2. Pick a short 30 fps clip with moderate motion, such as a person walking, and save it as video.mp4.

  3. Run python inference_video.py --exp=1 --video=video.mp4 to produce a 2x version at 60 fps.

  4. Run it again with --exp=2 to produce a 4x version for comparison.

  5. Step through the output frame by frame and mark where generated frames show ghosting, stretching or broken thin objects.

  6. Repeat with a clip that has a fast pan or a scene cut to see where the motion model fails.

Cosa mostra

The learner has smoother 60 fps and 120 fps versions of the clip and a list of failure cases. To check quality objectively, drop every second frame of a real 60 fps clip, interpolate it back and compare the generated frames with the held-out originals using PSNR or SSIM.

Implementazione nel mondo reale

NVIDIA's DLSS 3, announced in September 2022 for GeForce RTX 40 Series cards, generates an entire extra frame between rendered game frames using the previous and current frames, a hardware optical flow field and engine motion vectors.

Google Research's FILM, published at ECCV 2022, turns two near-duplicate photos into a short slow-motion clip and was released as open-source TensorFlow code under the Apache-2.0 license.

The open-source RIFE model is used inside tools such as Flowframes, SVP and Autodesk Flame to raise video frame rates, and its authors report more than 30 frames per second for doubling 720p video on an RTX 2080 Ti.

NVIDIA's Super SloMo, presented at CVPR 2018, was trained on 1,132 clips filmed at 240 frames per second and can produce any number of in-between frames, turning ordinary footage into slow motion.

FILM's command-line tool turns a folder of burst photos into a smooth clip: each extra pass doubles the in-between frames, giving (2^n + 1) x (frames - 1) output frames for n passes, written as images and an MP4.

Il futuro dell'interpolazione dei fotogrammi video

Two trends are visible in recent research. First, interpolation is converging with video generation. LDMVFI, accepted at AAAI 2024, treats interpolation as conditional generation with a latent diffusion model and reports better perceptual quality than earlier methods in quantitative tests and a user study. Second, real-time frame generation has moved into consumer products: NVIDIA's DLSS 3, announced in September 2022, generates whole frames in games on dedicated hardware. Open problems remain. Very large motion, occlusion and thin structures still cause visible errors, which is why FILM's 2022 paper focused on large-motion benchmarks. Evaluation is unsettled, since pixel metrics such as PSNR and L1 can disagree with human judgments of sharpness and plausibility, as the LDMVFI authors argue. Latency is a constraint for interactive use, because generating a frame between two rendered ones means waiting for the second. There are also questions of trust. As generated frames become harder to spot, footage used for journalism, science or legal evidence may need clear labeling of which frames were captured and which were synthesized. How such provenance will be recorded is still being worked out across the industry.

Metterlo in pratica

Perché è importante

  • Velocità e scala. L’intelligenza artificiale visiva può automatizzare le attività di ispezione, rilevamento ed etichettatura su larga scala.
  • Scelte di build. I team creativi possono prototipare i concetti più velocemente con meno revisioni manuali.
  • Team e flusso di lavoro. Le operazioni possono utilizzare segnali immagine e video che in precedenza erano difficili da elaborare.

Attenzione

  • I diritti di immagine e il consenso possono diventare rischi legali se la provenienza non è chiara.
  • Le prestazioni del modello possono variare in base all'illuminazione, ai dati demografici e agli ambienti.
  • I falsi positivi possono passare inosservati a meno che non vengano monitorate le soglie di confidenza.

Prossimi passi

  1. Definire i criteri di accettazione per i costi di precisione, richiamo ed errore.
  2. Testare con dati che corrispondono alle reali condizioni di produzione.
  3. Aggiungi la revisione umana per previsioni poco attendibili o ad alto impatto.
  4. Tieni traccia della deriva del modello e riconvalida dopo le modifiche alla fotocamera o al set di dati.

Fonti e approfondimenti

  1. arXiv (CVPR 2018)Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
  2. arXiv (ICCV 2017)Video Frame Interpolation via Adaptive Separable Convolution
  3. arXiv (ECCV 2022)Real-Time Intermediate Flow Estimation for Video Frame Interpolation
  4. GitHubECCV2022-RIFE repository
  5. arXiv (ECCV 2022)FILM: Frame Interpolation for Large Motion
  6. Google Research on GitHubFILM: Frame Interpolation for Large Motion (official code)
  7. NVIDIANVIDIA DLSS 3: AI-Powered Performance Multiplier Boosts Frame Rates By Up To 4X
  8. arXiv (AAAI 2024)LDMVFI: Video Frame Interpolation with Latent Diffusion Models

Continua a esplorare

Newsletter gratuita

Ricevi il briefing quotidiano sull'IA

Tre storie verificate sull'IA ogni mattina nei giorni feriali, scritte in un inglese semplice. Gratuito per sempre, senza pubblicità.

Un'email ogni giorno feriale. Disiscriviti con un clic. Non vendiamo né condividiamo mai il tuo indirizzo.

Mettiti alla prova

Take the Video Frame Interpolation quiz

Feedback immediato su ogni risposta e un certificato condivisibile con un ID verificabile una volta superato un corso.

Inizia il quiz

Sostieni l'istruzione gratuita sull'IA. AI Understanding è un'organizzazione non profit 501(c)(3) — nessun annuncio, nessun paywall, mai. Fai una donazione

Domande frequenti

What is Video Frame Interpolation?

Video frame interpolation is the task of creating new frames between two existing frames of a video, so footage can play more smoothly or in slow motion. Modern methods use neural networks that estimate how each pixel moves and then warp and blend the neighboring frames. The same idea powers slow-motion tools, frame-rate converters and frame generation in games.

What does video frame interpolation add to a video clip?

L'interpolazione dei fotogrammi sintetizza fotogrammi intermedi plausibili tra quelli esistenti, uniformando il movimento o consentendo la riproduzione al rallentatore.

Perché l'interpolazione non può semplicemente mediare (fondere) due fotogrammi adiacenti?

Poiché gli oggetti si spostano tra i fotogrammi, la fusione ingenua crea immagini fantasma; gli interpolatori devono tenere conto del movimento per posizionare correttamente gli oggetti nel tempo intermedio.

Cos'è il flusso ottico nel contesto dell'interpolazione dei frame?

Il flusso ottico stima il movimento di ciascun pixel tra i fotogrammi, consentendo al modello di deformare il contenuto nella posizione intermedia corretta.

Perché modelli come DAIN incorporano informazioni di profondità?

La consapevolezza della profondità consente al modello di risolvere l'occlusione durante la deformazione, in modo che gli oggetti più vicini si sovrappongano correttamente a quelli distanti, riducendo gli artefatti.

Cosa si intende per "disocclusione" come sfida nell'interpolazione?

Man mano che gli oggetti si muovono, lo sfondo precedentemente nascosto diventa visibile e deve essere sintetizzato, poiché non era completamente presente in nessuno dei fotogrammi di input.