視覺人工智慧指南

新穎的視圖合成

新視角合成是指利用一組現有照片,渲染出從從未拍攝過的相機位置所看到的場景樣貌。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 捕捉並檢查一個小型高斯點雲場景
  5. 現實世界的實施
  6. 新穎視圖合成的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

神經輻照度場(NeRF,2020)和 3D 高斯點雲(2023)將其從一個研究奇觀轉變為一種實際方法,可用於從普通照片構建可探索的 3D 場景,且較新的方法能以 1080p 解析度進行即時渲染。

重點摘要

  1. 新視角合成利用帶有位姿的輸入照片和擬合的 3D 表示,從從未拍攝過的相機位置渲染場景。
  2. NeRF 於 2020 年推出,它將場景儲存在神經網路中,並沿著相機射線進行查詢,在其合成基準測試中取得了 31.01 dB 的 PSNR 分數。
  3. 原始 NeRF 每個場景需要一到兩天的訓練時間,並使用 V100 GPU 渲染每一幀大約需要 30 秒。
  4. 3D 高斯點雲於 2023 年發表,能以每秒約 134 幀的速度渲染無界真實場景,且品質接近 Mip-NeRF 360。
  5. 點雲技術以記憶體換取速度,每個場景使用數百 MB 的記憶體,而 NeRF 網路僅需幾 MB。
  6. 輸入照片從未拍攝到的區域無法被重建,只能被推測,這就是為什麼單圖像方法依賴生成模型。

深入探討

Novel view synthesis means producing a new image of a scene from a viewpoint where no camera stood. The input is a collection of photos, plus an estimate of where each camera was and which way it pointed, called its pose. The output is a renderer: give it any new pose and it returns the image a camera there would have taken. If a method only blends nearby photos, views far from the originals break down. Modern methods therefore build an internal 3D representation of the scene, fit it so that rendering from the known poses reproduces the input photos, and then render from new poses. What the model never saw, such as the back of a statue photographed only from the front, has to be guessed or left blank.

Almost every pipeline starts with structure-from-motion (SfM), a classical computer vision step that matches feature points across photos and solves for the camera poses and a sparse cloud of 3D points. Google's 2023 Research post on indoor reconstructions for Maps describes exactly this step: photos are processed with structure-from-motion to recover each camera's position, orientation and lens properties before any neural model is trained. Errors here, such as a wrongly placed camera, cause blur and ghosting later, because the model tries to reconcile photos that do not actually agree.

The neural radiance field, or NeRF, was introduced in March 2020 by Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi and Ren Ng, and presented as an oral at ECCV 2020. A NeRF is a fully connected neural network that takes a 3D position and a viewing direction and returns two things: a density, meaning how much that point blocks light, and a color as seen from that direction. To render a pixel, the method shoots a ray from the camera through the pixel, samples points along it, queries the network at each point and adds up the colors weighted by how much light survives to reach the camera. That process, called volume rendering, is differentiable, so the network can be trained with gradient descent simply by comparing rendered pixels with the real photos.

NeRF's results were a step change. On the paper's synthetic 360-degree scenes, built from 100 input views each, it scored 31.01 dB peak signal-to-noise ratio (PSNR, a pixel-accuracy measure where higher is better) against 26.05 dB for the best prior method, Neural Volumes. On real forward-facing scenes it scored 26.50 dB against 24.13 dB for LLFF. The whole scene fit in about 5 MB of network weights, compared with over 15 GB for LLFF on one synthetic scene. The costs were speed: training took 100,000 to 300,000 iterations, about one to two days on an NVIDIA V100 GPU, and rendering a single frame took about 30 seconds.

Three lines of work addressed those costs and limits. Instant-NGP, from Thomas Müller and colleagues at NVIDIA in January 2022 and published at SIGGRAPH 2022, replaced most of the large network with a multiresolution hash table of trainable features, cutting training to seconds or minutes and rendering 1920 by 1080 images in tens of milliseconds. Mip-NeRF 360, from Jonathan Barron and colleagues in November 2021, handled unbounded scenes where the camera circles a subject and the background stretches to the horizon, reporting 57% lower mean squared error than its predecessor mip-NeRF. Then 3D Gaussian splatting, by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis, published in ACM Transactions on Graphics in July 2023, dropped the neural network entirely. It represents the scene as millions of small, semi-transparent, stretched 3D blobs called Gaussians, each with a position, shape, opacity and view-dependent color, and draws them with a fast rasterizer instead of tracing rays.

The trade-offs show up clearly in the splatting paper's comparison on the Mip-NeRF 360 dataset. Mip-NeRF 360 reached 27.69 dB PSNR but needed 48 GPU-hours of training and rendered at 0.06 frames per second. Instant-NGP trained in under 8 minutes and rendered at about 10 frames per second, but scored 25.3 to 25.6 dB. Gaussian splatting reached 27.21 dB after 41 minutes of training, rendered at 134 frames per second on an NVIDIA A6000, and had better SSIM and LPIPS perceptual scores than Mip-NeRF 360. Its price is memory: 734 MB per scene against 8.6 MB for Mip-NeRF 360, with 1 to 5 million Gaussians in real scenes.

A separate branch tackles the hardest version of the problem, novel views from one or a few images. Zero-1-to-3, posted in March 2023 by Ruoshi Liu, Carl Vondrick and colleagues, fine-tunes a large image diffusion model on synthetic objects so that it can generate an object from a requested camera rotation given a single photo. It generalizes to real photos and even paintings, but the unseen side is invented from learned priors, not measured. Per-scene methods such as NeRF and splatting instead reconstruct only what the photos support.

Failure modes are consistent across methods. The splatting authors list artifacts in poorly observed regions, elongated or splotchy Gaussians, and popping when large Gaussians appear. NeRF-style models assume a static scene with constant lighting; Google's indoor pipeline adds per-image appearance codes from NeRF in the Wild and exposure inputs from Block-NeRF to cope with changing light, and its post lists dim bars as a hard case. Moving people, reflections, thin structures and blank walls with few features all degrade results. In practice a usable capture needs dozens to hundreds of sharp, overlapping photos with fixed exposure settings, a GPU for training, and, for splatting, a viewer that can hold hundreds of megabytes per scene.

04工作範例

捕捉並檢查一個小型高斯點雲場景

  1. 選擇一個靜態物件或房間,鎖定相機的曝光與白平衡,並在兩個高度繞完整圓走動,拍攝約 100 到 200 張銳利照片。

  2. 對照片執行結構光束恢復(structure-from-motion)軟體,以估算相機姿態與稀疏點雲,並剔除無法註冊的照片。

  3. 將每第八張照片保留作為測試視圖,模型在訓練時永不使用。

  4. 在剩餘的照片上訓練 3D Gaussian splatting 模型,從稀疏點開始,執行標準的 30,000 次迭代。

  5. 渲染保留的測試視圖,計算與真實照片的 PSNR,然後檢查最差的影像以找出漂浮物與模糊。

  6. 在檢視器中環繞場景,留意偽影出現的位置,通常在照片覆蓋較少的區域。

它顯示了什麼

一個互動式 3D 場景加上測量的測試分數。真實場景的 PSNR 在 20 多到高 20 之間,屬於 2023 年 splatting 論文報告的範圍;若分數顯著較低,通常代表姿態錯誤、運動模糊或捕捉時光線變化。

現實世界的實施

Google Maps 的 Immersive View 以 DSLR 照片建構室內餐廳與咖啡廳的視圖,使用基於 mip-NeRF 360 的模型,並以預設飛行路徑以 360 度影片形式呈現,資訊來自 2023 年 Google Research 的貼文。

將房間或物件的環繞手機或相機拍攝轉換為可互動的 3D Gaussian splat,能以 1080p 及每秒超過 30 幀的速度環繞檢視,這是 2023 年 splatting 論文設定的目標。

使用 NVIDIA 的 Instant‑NGP,在數分鐘而非數天內訓練輻射場,其 2022 年論文報告在訓練數秒後即可得到高品質結果,得益於融合的 CUDA 核心。

使用 Zero‑1‑to‑3,這個 2023 年的擴散模型以相對相機旋轉為條件,從單張照片生成合理的側面與背面視圖,作為單圖像 3D 重建的起點。

在 Mip‑NeRF 360 的無界真實場景資料集上比較新方法,該基準中 3D Gaussian splatting 報告 27.21 dB PSNR,對比 Mip‑NeRF 360 的 27.69 dB。

新穎視圖合成的未來

Several open problems stand out in the primary literature. The 2023 Gaussian splatting paper names memory as a cost, with hundreds of megabytes per scene and training peaks above 20 GB for large scenes, so compression and streaming are active topics. It also lists artifacts in poorly observed regions and popping during camera motion, which points to better regularization and visibility handling. Google's 2023 Research post on indoor reconstructions names real-time on-device neural rendering, scene segmentation for search and navigation, and outdoor photo collections as open questions, and it chose pre-rendered videos over live rendering because real-time inference was too costly for phones. Dynamic scenes with moving people and changing light remain harder than static ones. Few-view and single-view synthesis, illustrated by Zero-1-to-3 in 2023, depends on generative priors that invent unseen content, so measuring whether generated views are faithful rather than merely plausible is still unsettled. Whether radiance fields, splats or generative models will dominate particular applications is an open question.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXiv (ECCV 2020)NeRF:以神經輻射場表示場景以進行視角合成
  2. arXiv (ACM Transactions on Graphics, 2023)用於即時輻射場渲染的 3D 高斯噴射
  3. arXiv (SIGGRAPH 2022)具有多重解析度雜湊編碼的即時神經圖形基元
  4. arXivMip-NeRF 360:無界抗鋸齒神經輻射場
  5. arXiv零一到三:零樣本一張影像到 3D 對象
  6. Google Research用 NeRF 重建室內空間

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加小說觀點綜合測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是新視圖合成?

新穎的視圖合成是使用一組現有照片從未拍攝過的相機位置渲染場景的任務。神經輻射場(NeRF,2020)和 3D 高斯分佈(2023)將其從一種研究好奇心轉變為一種從普通圖片構建可探索的 3D 場景的實用方法,並使用更新的方法以 1080p 實時渲染。

給定一個場景的照片,新穎的視圖合成試圖產生什麼?

新穎的視圖合成使用一組具有已知姿勢的輸入影像,從新的、看不見的視點生成逼真的影像。

神經輻射場 (NeRF) 將 3D 位置和觀看方向映射到什麼?

NeRF 學習從(位置、方向)到發射顏色和密度的函數,然後沿著光線積分以渲染影像。

NeRF 如何為新視圖渲染像素?

對於每個像素,NeRF 投射一條射線,對 3D 點進行取樣,查詢網路的顏色/密度,並將它們與體積渲染積分進行合成。

3D Gaussian Splatting 的場景表示與原始 NeRF 風格的每光線網絡評估有什麼區別?

表示和渲染計算不同;速度和記憶體取決於它們的實現和場景。

為什麼 NVS 方法可以再現反射和閃亮的高光?

透過根據觀察方向調節顏色,NeRF 和高斯斑點可捕捉依賴視圖的效果,例如鏡面高光和反射。