Vizuális MI ÚTMUTATÓ

Regénynézet szintézis

Novel view synthesis is the task of rendering what a scene would look like from a camera position that was never photographed, using a set of existing photos.

Hallgassa meg ezt az útmutatótAI-narrált · ~8 min

Ezen az oldalon9 perc olvasás
  1. Áttekintés
  2. Főbb tanulságok
  3. Mély merülés
  4. Capturing and checking a small Gaussian splat scene
  5. Valós megvalósítás
  6. A regénynézeti szintézis jövője
  7. A gyakorlatba ültetés
  8. Források és további olvasmányok
  9. Folytassa a felfedezést
  10. Gyakran ismételt kérdések

Áttekintés

Neural radiance fields (NeRF, 2020) and 3D Gaussian splatting (2023) turned it from a research curiosity into a practical way to build explorable 3D scenes from ordinary pictures, with the newer method rendering at 1080p in real time.

Főbb tanulságok

  1. Novel view synthesis renders a scene from camera positions that were never photographed, using posed input photos and a fitted 3D representation.
  2. NeRF, introduced in 2020, stores a scene in a neural network queried along camera rays and scored 31.01 dB PSNR on its synthetic benchmark.
  3. Original NeRF needed one to two days of training per scene and about 30 seconds to render each frame on a V100 GPU.
  4. 3D Gaussian splatting, published in 2023, renders unbounded real scenes at about 134 frames per second with quality close to Mip-NeRF 360.
  5. Splatting trades speed for memory, using hundreds of megabytes per scene against a few megabytes for a NeRF network.
  6. Regions the input photos never saw cannot be reconstructed, only guessed, which is why single-image methods rely on generative models.

Mély merülés

Novel view synthesis means producing a new image of a scene from a viewpoint where no camera stood. The input is a collection of photos, plus an estimate of where each camera was and which way it pointed, called its pose. The output is a renderer: give it any new pose and it returns the image a camera there would have taken. If a method only blends nearby photos, views far from the originals break down. Modern methods therefore build an internal 3D representation of the scene, fit it so that rendering from the known poses reproduces the input photos, and then render from new poses. What the model never saw, such as the back of a statue photographed only from the front, has to be guessed or left blank.

Almost every pipeline starts with structure-from-motion (SfM), a classical computer vision step that matches feature points across photos and solves for the camera poses and a sparse cloud of 3D points. Google's 2023 Research post on indoor reconstructions for Maps describes exactly this step: photos are processed with structure-from-motion to recover each camera's position, orientation and lens properties before any neural model is trained. Errors here, such as a wrongly placed camera, cause blur and ghosting later, because the model tries to reconcile photos that do not actually agree.

The neural radiance field, or NeRF, was introduced in March 2020 by Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi and Ren Ng, and presented as an oral at ECCV 2020. A NeRF is a fully connected neural network that takes a 3D position and a viewing direction and returns two things: a density, meaning how much that point blocks light, and a color as seen from that direction. To render a pixel, the method shoots a ray from the camera through the pixel, samples points along it, queries the network at each point and adds up the colors weighted by how much light survives to reach the camera. That process, called volume rendering, is differentiable, so the network can be trained with gradient descent simply by comparing rendered pixels with the real photos.

NeRF's results were a step change. On the paper's synthetic 360-degree scenes, built from 100 input views each, it scored 31.01 dB peak signal-to-noise ratio (PSNR, a pixel-accuracy measure where higher is better) against 26.05 dB for the best prior method, Neural Volumes. On real forward-facing scenes it scored 26.50 dB against 24.13 dB for LLFF. The whole scene fit in about 5 MB of network weights, compared with over 15 GB for LLFF on one synthetic scene. The costs were speed: training took 100,000 to 300,000 iterations, about one to two days on an NVIDIA V100 GPU, and rendering a single frame took about 30 seconds.

Three lines of work addressed those costs and limits. Instant-NGP, from Thomas Müller and colleagues at NVIDIA in January 2022 and published at SIGGRAPH 2022, replaced most of the large network with a multiresolution hash table of trainable features, cutting training to seconds or minutes and rendering 1920 by 1080 images in tens of milliseconds. Mip-NeRF 360, from Jonathan Barron and colleagues in November 2021, handled unbounded scenes where the camera circles a subject and the background stretches to the horizon, reporting 57% lower mean squared error than its predecessor mip-NeRF. Then 3D Gaussian splatting, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis, published in ACM Transactions on Graphics in July 2023, dropped the neural network entirely. It represents the scene as millions of small, semi-transparent, stretched 3D blobs called Gaussians, each with a position, shape, opacity and view-dependent color, and draws them with a fast rasterizer instead of tracing rays.

The trade-offs show up clearly in the splatting paper's comparison on the Mip-NeRF 360 dataset. Mip-NeRF 360 reached 27.69 dB PSNR but needed 48 GPU-hours of training and rendered at 0.06 frames per second. Instant-NGP trained in under 8 minutes and rendered at about 10 frames per second, but scored 25.3 to 25.6 dB. Gaussian splatting reached 27.21 dB after 41 minutes of training, rendered at 134 frames per second on an NVIDIA A6000, and had better SSIM and LPIPS perceptual scores than Mip-NeRF 360. Its price is memory: 734 MB per scene against 8.6 MB for Mip-NeRF 360, with 1 to 5 million Gaussians in real scenes.

A separate branch tackles the hardest version of the problem, novel views from one or a few images. Zero-1-to-3, posted in March 2023 by Ruoshi Liu, Carl Vondrick and colleagues, fine-tunes a large image diffusion model on synthetic objects so that it can generate an object from a requested camera rotation given a single photo. It generalizes to real photos and even paintings, but the unseen side is invented from learned priors, not measured. Per-scene methods such as NeRF and splatting instead reconstruct only what the photos support.

Failure modes are consistent across methods. The splatting authors list artifacts in poorly observed regions, elongated or splotchy Gaussians, and popping when large Gaussians appear. NeRF-style models assume a static scene with constant lighting; Google's indoor pipeline adds per-image appearance codes from NeRF in the Wild and exposure inputs from Block-NeRF to cope with changing light, and its post lists dim bars as a hard case. Moving people, reflections, thin structures and blank walls with few features all degrade results. In practice a usable capture needs dozens to hundreds of sharp, overlapping photos with fixed exposure settings, a GPU for training, and, for splatting, a viewer that can hold hundreds of megabytes per scene.

04Működő példa

Capturing and checking a small Gaussian splat scene

  1. Pick a static object or room, lock the camera's exposure and white balance, and take about 100 to 200 sharp photos while walking a full circle at two heights.

  2. Run structure-from-motion software on the photos to estimate camera poses and a sparse point cloud, and discard photos that fail to register.

  3. Set aside every eighth photo as a test view that the model will never train on.

  4. Train a 3D Gaussian splatting model on the remaining photos, starting from the sparse points, for the standard 30,000 iterations.

  5. Render the held-out test views and compute PSNR against the real photos, then look at the worst images for floaters and blur.

  6. Orbit the scene in a viewer and note where artifacts appear, usually in areas that few photos covered.

Mit mutat

An interactive 3D scene plus a measured test score. PSNR in the mid to high 20s on real scenes is in the range the 2023 splatting paper reports; much lower scores usually mean bad poses, motion blur or changing light during capture.

Valós megvalósítás

Google Maps' Immersive View shows indoor views of restaurants and cafes built from DSLR photos with a mip-NeRF 360 based model, delivered as 360-degree videos along preset flight paths, according to a 2023 Google Research post.

Turning a walk-around phone or camera capture of a room or object into an interactive 3D Gaussian splat that can be orbited at more than 30 frames per second at 1080p, the target set by the 2023 splatting paper.

Training a radiance field in minutes rather than days with NVIDIA's Instant-NGP, whose 2022 paper reports high-quality results after a matter of seconds of training using fused CUDA kernels.

Generating plausible side and back views of an object from a single photo with Zero-1-to-3, a 2023 diffusion model conditioned on a relative camera rotation, as a starting point for single-image 3D reconstruction.

Comparing new methods on the Mip-NeRF 360 dataset of unbounded real scenes, the benchmark where 3D Gaussian splatting reported 27.21 dB PSNR against 27.69 dB for Mip-NeRF 360.

A regénynézeti szintézis jövője

Several open problems stand out in the primary literature. The 2023 Gaussian splatting paper names memory as a cost, with hundreds of megabytes per scene and training peaks above 20 GB for large scenes, so compression and streaming are active topics. It also lists artifacts in poorly observed regions and popping during camera motion, which points to better regularization and visibility handling. Google's 2023 Research post on indoor reconstructions names real-time on-device neural rendering, scene segmentation for search and navigation, and outdoor photo collections as open questions, and it chose pre-rendered videos over live rendering because real-time inference was too costly for phones. Dynamic scenes with moving people and changing light remain harder than static ones. Few-view and single-view synthesis, illustrated by Zero-1-to-3 in 2023, depends on generative priors that invent unseen content, so measuring whether generated views are faithful rather than merely plausible is still unsettled. Whether radiance fields, splats or generative models will dominate particular applications is an open question.

A gyakorlatba ültetés

Miért számít

  • Sebesség és méretarány. A vizuális AI képes automatizálni az ellenőrzési, észlelési és címkézési feladatokat nagy léptékben.
  • Építési lehetőségek. A kreatív csapatok gyorsabban prototípusokat készíthetnek a koncepciókból, kevesebb kézi átdolgozással.
  • Csapat és munkafolyamat. A műveletek olyan kép- és videojeleket használhatnak, amelyeket korábban nehéz volt feldolgozni.

Vigyázz

  • A képhez fűződő jogok és a beleegyezés jogi kockázatot jelenthet, ha a származás nem egyértelmű.
  • A modell teljesítménye a világítástól, a demográfiai adatoktól és a környezettől függően változhat.
  • A hamis pozitívumok észrevétlenek maradhatnak, hacsak nem figyelik a megbízhatósági küszöböket.

Következő lépések

  1. Határozza meg a pontosság, a visszahívás és a hibaköltségek elfogadási kritériumait.
  2. Tesztelje a valós gyártási feltételeknek megfelelő adatokkal.
  3. Adjon hozzá emberi felülvizsgálatot az alacsony megbízhatóságú vagy nagy hatású előrejelzésekhez.
  4. A modell elsodródásának nyomon követése és újbóli érvényesítése a kamera vagy az adatkészlet módosítása után.

Források és további olvasmányok

  1. arXiv (ECCV 2020)NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
  2. arXiv (ACM Transactions on Graphics, 2023)3D Gaussian Splatting for Real-Time Radiance Field Rendering
  3. arXiv (SIGGRAPH 2022)Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
  4. arXivMip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields
  5. arXivZero-1-to-3: Zero-shot One Image to 3D Object
  6. Google ResearchReconstructing indoor spaces with NeRF

Folytassa a felfedezést

Ingyenes hírlevél

Szerezze meg a napi AI összefoglalót

Minden hétköznap reggel három ellenőrzött AI történet, egyszerű angol nyelven. Örökké ingyen, hirdetések nélkül.

Minden hétköznap egy e-mail. Egy kattintással lemondhat. Soha nem értékesítjük vagy osztjuk meg az Ön címét.

Teszteld magad

Take the Novel View Synthesis quiz

Azonnali visszajelzés minden válaszra, és megosztható tanúsítvány egy ellenőrizhető azonosítóval, miután sikeresen teljesít egy kurzust.

Kezdő kvíz

Támogassa az ingyenes AI oktatást. AI Understanding egy 501(c)(3) nonprofit – nincs hirdetés, nincs fizetőfal, soha. Tegyen adományt

Gyakran ismételt kérdések

Mi az a Novel View szintézis?

Novel view synthesis is the task of rendering what a scene would look like from a camera position that was never photographed, using a set of existing photos. Neural radiance fields (NeRF, 2020) and 3D Gaussian splatting (2023) turned it from a research curiosity into a practical way to build explorable 3D scenes from ordinary pictures, with the newer method rendering at 1080p in real time.

Egy jelenetről készült fényképek alapján mit próbál előállítani az újszerű nézetszintézis?

Az újszerű nézetszintézis fotorealisztikus képeket hoz létre új, nem látott nézőpontokból, ismert pózokkal rendelkező bemeneti képek segítségével.

Mire képezi le a neurális sugárzási mező (Neural Radiance Field, NeRF) a 3D pozíciót és a nézési irányt?

A NeRF megtanul egy függvényt (pozíciótól, iránytól) a kibocsátott színig és sűrűségig, amelyet aztán a sugarak mentén integrálva jelenít meg képeket.

Hogyan jelenít meg a NeRF egy pixelt egy új nézethez?

A NeRF minden pixelre egy sugarat bocsát ki, 3D-s pontokat vesz mintát, lekérdezi a hálózattól a színt/sűrűséget, és összeállítja azokat a térfogat-renderelő integrálóval.

What distinguishes 3D Gaussian Splatting's scene representation from an original NeRF-style per-ray network evaluation?

The representations and rendering computations differ; speed and memory depend on their implementation and scene.

Miért képesek az NVS-módszerek a tükröződéseket és a fényes csúcsokat reprodukálni?

A színeknek a nézési irányhoz való kondicionálásával a NeRF és Gauss jelek olyan nézetfüggő hatásokat rögzítenek, mint a tükörfények és a tükröződések.