視覺人工智慧指南

DMTet:3D 形狀的深度行進四面體

DMTet,深度行進四面體的簡稱,是由 NVIDIA 研究人員於 NeurIPS 2021 年提出的 3D 形狀表示方法

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 在單個單元上追踪行進四面體
  5. 現實世界的實施
  6. DMTet 的未來:3D 形狀的深度行進四面體
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

它在可變形四面體網格上存儲帶符號距離場,並通過可微步驟將其轉換為三角網格,從而使神經網絡能直接在最終表面上訓練。它已成為生成和重建 3D 網格的標準構件

重點摘要

  1. DMTet 在可變形四面體網格上存儲帶符號距離值和頂點偏移,並使用行進四面體提取三角網格
  2. 行進四面體每個四面體有 16 種符號模式,通過對稱可歸約為三種不同的表面情況
  3. 由於提取過程可微,DMTet 可在最終網格上測量損失(如 Chamfer 距離和對抗損失)進行訓練
  4. 選擇性體積細分使成本隨表面積而非體積增長,隨著解析度提高
  5. NVIDIA 在其開源 Kaolin 庫中發布了 DMTet 的核心操作,GET3D 使用 DMTet 提取網格

深入探討

DMTet is a way of describing a 3D shape so that a neural network can both learn it easily and output it as a usable mesh. A mesh is the list of triangles that games, film tools and 3D printers expect. Neural networks find meshes hard to produce directly because the number of triangles and the shape's topology, meaning how many holes and separate pieces it has, must be fixed in advance. DMTet avoids that by learning a signed distance field (SDF), a function that gives every point in space its distance to the surface, negative inside the object and positive outside, and then extracting the mesh where that value crosses zero. Because the extraction step is differentiable, errors measured on the final mesh can be sent back to adjust the field. The paper calls this a hybrid of implicit and explicit representations.

The structure underneath is a tetrahedral grid. A tetrahedron is a pyramid with four triangular faces, and a grid of them can fill a cube completely, much like cubes fill a voxel grid. DMTet stores one SDF value at every corner, or vertex, of this grid, plus a small learned offset that lets each vertex move. The paper adopts this deformable grid from an earlier NVIDIA method called DefTet, but stores signed distances on vertices instead of occupancy on cells, which gives finer control over where the surface lies. Moving vertices lets the grid line up with thin parts such as chair legs or lamp stems without raising the overall resolution.

Turning the field into triangles uses marching tetrahedra, a cousin of the marching cubes algorithm. For each tetrahedron, the method checks the signs of its four corner values. Four signs give 16 combinations, which the paper notes reduce to three distinct cases by symmetry: no surface, one triangle or two triangles. Where an edge connects a negative corner to a positive one, the surface vertex is placed by linear interpolation at the point where the value would reach zero. Earlier work had argued that this formula blocks changes in topology during training because it divides by the difference of two values. The DMTet authors point out that the formula is only evaluated on edges whose corners have opposite signs, so the division never fails and gradients flow to both positions and SDF values.

The full network runs coarse to fine. An encoder called PVCNN turns the input, a point cloud or points sampled from a coarse voxel shape, into a 3D feature volume. A small fully connected network predicts an initial SDF value and feature for every grid vertex. The method then finds the surface tetrahedra, those whose corners disagree in sign, and a graph convolutional network predicts position offsets and SDF corrections for their vertices. Next comes volume subdivision: each surface tetrahedron and its neighbors are split into eight smaller ones by adding edge midpoints, and the rest of the grid is dropped. The paper notes that this keeps cost growing roughly with surface area, quadratically with resolution, rather than with volume. After extraction, a learnable version of Loop subdivision smooths and refines the mesh further.

Training compares the generated mesh to the target mesh. Points are sampled from both surfaces and the loss combines Chamfer distance, which measures how far each sampled point is from its nearest neighbor on the other surface, a normal consistency term, an adversarial loss from a 3D discriminator that judges local patches of the signed distance field, an SDF regularizer that keeps inside and outside consistent, and a penalty on large vertex movements. The authors report that DMTet beat ConvONet, an implicit method, and DECOR-GAN on their animal upsampling benchmark across all metrics, and that it ran about 10 times faster at inference than implicit-representation baselines. In their ablation, adding volume subdivision won 78% of human comparisons for overall look, and adding surface subdivision on top won 62%.

The obvious alternatives each lose something. Voxel grids are simple but their memory cost grows with the cube of resolution. Pure implicit networks, which predict an SDF or occupancy value for any queried point, handle any topology but are trained on sampled values rather than on the surface itself, and they need a separate and costly marching cubes pass to produce a mesh. Mesh-deformation methods that bend a template sphere give clean meshes but cannot add holes or separate parts. DMTet keeps arbitrary topology while supervising the final surface. In the paper's oracle test on ShapeNet chairs, marching tetrahedra captured thin structures better than marching cubes for the same number of queried points.

DMTet has known limits. Losses on the surface only reach vertices near the surface, so the paper needs an extra SDF loss to stop disconnected fragments from appearing. Flipping all signs in a tetrahedron yields the same triangles, so the surface alone cannot say which side is inside. Resolution is still bounded by the grid, and mesh quality depends on the number of subdivision levels. Later work such as FlexiCubes, published at SIGGRAPH 2023, argues that isosurface methods adapted from marching cubes or dual contouring lack the freedom to represent sharp, feature-preserving meshes during optimization or can be numerically unstable, and adds extra learnable parameters to address that.

In practice, DMTet needs a GPU, a tetrahedral grid file and a differentiable renderer or 3D loss. The paper's point cloud experiments report inference times on a single NVIDIA V100 GPU, and its base model uses a uniform grid of resolution 100. NVIDIA released the marching tetrahedra, volume subdivision and surface subdivision operations in its open-source Kaolin library, which is the usual starting point. Training data still matters: the animal results relied on 1,562 licensed models, and generators built on DMTet, such as GET3D, inherit the biases of their training collections.

04工作範例

在單個單元上追踪行進四面體

  1. 取一個四面體,頂點為 A、B、C、D,SDF 值分別為 -0.2、0.3、0.4、0.1。

  2. 標記符號:A 為內部(負值),B、C、D 為外部(正值),因此這是單角案例,產生一個三角形。

  3. 列出符號變化的邊:A 到 B、A 到 C、A 到 D。

  4. 在邊 A 到 B 上,將頂點放置在距離 A 到 B 的 0.2 / (0.2 + 0.3) = 0.4 處。

  5. 在邊 A 到 C 上,該比例為 0.2 / 0.6,約為 0.33;而在邊 A 到 D 上,該比例為 0.2 / 0.3,約為 0.67。

  6. 連接這三個點以形成一個三角形,其方向使其法線從 A 指向正角方向。

  7. 將 A 的值更改為 -0.25 並重新計算:每個頂點都會稍微向 B、C 和 D 滑動,這就是訓練損失所使用的梯度。

它顯示了什麼

學習者最後會得到一個表面三角形,以及對於改變 SDF 值如何平滑移動網格的直觀理解。可以使用 Kaolin 在單四面體網格上使用相同值的 marching_tetrahedra 函式來進行檢查。

現實世界的實施

原始的 DMTet 論文將粗糙的 16 x 16 x 16 體素動物(類似於方塊遊戲模型)上採樣為詳細的網格,並在 1,562 個授權的 TurboSquid 動物模型上進行訓練,其中 1,120 個用於訓練,442 個用於測試。

NVIDIA 的 GET3D 生成器於 NeurIPS 2022 發表,它從 2D 圖像集合中學習,並使用 DMTet 從其預測的有號距離場中提取表面網格,從而生成帶紋理的汽車、椅子、動物和角色。

NVIDIA 的 nvdiffrec 方法使用可微分的 marching tetrahedra 從多視角照片中恢復三角形網格、材質與光照,所產生的資產能在標準的基於三角形的引擎中以互動速率進行渲染。

NVIDIA 的 Kaolin 程式庫附帶了一個教學:「使用 DMTet 重建點雲」(Reconstructing Point Cloud with DMTet),該教學訓練了一個有號距離估計器,利用 Chamfer 距離損失與 marching tetrahedra 從點雲重建網格。

在該論文的 ShapeNet 評測基準中,DMTet 從涵蓋 13 個物件類別(包括椅子、燈具和飛機)的 5,000 個點之雜訊點雲中重建網格。

DMTet 的未來:3D 形狀的深度行進四面體

DMTet's main legacy is as a component. NVIDIA's GET3D (NeurIPS 2022) uses it to turn predicted signed distance fields into textured meshes trained only from 2D images, and nvdiffrec, first posted in November 2021, uses differentiable marching tetrahedra to recover meshes, materials and lighting from photos. These show a pattern in which a differentiable mesh extractor sits between a learned field and a rasterizer, so that 2D image losses can shape 3D geometry. Open problems center on mesh quality and resolution. FlexiCubes (SIGGRAPH 2023) argues that extraction layers adapted from marching cubes or dual contouring lack the degrees of freedom to keep sharp features and can be unstable during optimization, and proposes extra learnable parameters on top of a dual marching cubes grid. Researchers continue to compare tetrahedral and cubic grids for thin structures, memory use and triangle quality. Another direction is scale. DMTet was evaluated on curated sets, about 1,500 licensed animal models and 13 ShapeNet categories, and its quality outside such domains depends on the data. How well grid-based extractors handle open surfaces, very thin sheets and scenes rather than single objects remains an active question.

付諸實踐

為什麼這很重要

  • 速度與規模. 視覺人工智慧可以大規模自動化檢查、檢測和標記任務。
  • 配裝選擇. 創意團隊可以透過更少的手動修改來更快地建立概念原型。
  • 團隊與工作流程. 操作可以使用以前難以處理的影像和視訊訊號。

留意

  • 如果出處不明,肖像權和同意可能會成為法律風險。
  • 模型表現可能因光照、人口統計和環境的不同而有所不同。
  • 除非監控置信閾值,否則誤報可能會被忽略。

後續步驟

  1. 定義精確度、召回率和錯誤成本的接受標準。
  2. 使用符合實際生產條件的數據進行測試。
  3. 為低置信度或高影響力的預測添加人工審核。
  4. 追蹤模型漂移並在相機或資料集變更後重新驗證。

資料來源與延伸閱讀

  1. arXivDeep Marching Tetrahedra:用於高解析度 3D 形狀合成的混合表示法
  2. NeurIPS 2021 ProceedingsDeep Marching Tetrahedra:用於高解析度 3D 形狀合成的混合表示法(論文 PDF)
  3. NVIDIA Toronto AI LabDMTet 專案頁面
  4. NVIDIA Kaolin documentation教學索引(使用 DMTet 重建點雲)
  5. NVIDIA Toronto AI LabGET3D:從圖像學習的高品質 3D 紋理形狀生成模型
  6. arXiv從圖像中提取三角形 3D 模型、材質與光照
  7. arXiv (SIGGRAPH 2023)用於基於梯度的網格優化的靈活等值面提取

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

DMTet 測驗:用於 3D 形狀的 Deep Marching Tetrahedra

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是 DMTet:用於 3D 形狀的 Deep Marching Tetrahedra?

DMTet(全名為 Deep Marching Tetrahedra)是由 NVIDIA 研究人員於 NeurIPS 2021 推出的 3D 形狀表示法。它將有號距離場儲存於可變形的四面體網格上,並透過可微分的步驟將其轉化為三角形網格,因此神經網路可以直接在最終的表面上進行訓練。它已成為生成與重建 3D 網格的標準建構區塊。

DMTet 在其混合表示中結合了哪兩種成分?

DMTet 將可變形四面體網格與預測的有符號距離值配對,以實現可微網格提取。

每個頂點的有符號距離值表示什麼?

帶符號的距離場以表示內部與外部的符號對到表面的距離進行編碼,因此過零定義了表面。

DMTet 中如何擷取顯式三角形網格?

可微分行進四面體步驟將曲面頂點放置在 SDF 跨越四面體邊緣改變符號的位置。

儘管拓樸發生變化,是什麼使得 DMTet 的表面提取仍可微分?

儘管拓撲變化是離散的,但梯度會透過連續的 SDF 值傳播並學習設定表面頂點位置的頂點變形。

DMTet 如何有效地添加精細的幾何細節?

它只細分靠近表面的四面體,將容量集中在需要細節的地方,而不是空白的空間。