Visuele AI-GIDS
DMTet: Deep Marching Tetrahedra for 3D Shapes
DMTet, short for Deep Marching Tetrahedra, is a 3D shape representation introduced by NVIDIA researchers at NeurIPS 2021.
Luister naar deze gidsAI-verteld · ~8 min
Op deze pagina9 minuten lezen
Overzicht
It stores a signed distance field on a deformable grid of tetrahedra and turns it into a triangle mesh with a differentiable step, so a neural network can be trained directly on the final surface. It became a standard building block for generating and reconstructing 3D meshes.
Belangrijkste inzichten
- DMTet stores signed distance values and vertex offsets on a deformable tetrahedral grid and extracts a triangle mesh with marching tetrahedra.
- Marching tetrahedra has 16 sign patterns per tetrahedron, which reduce to three distinct surface cases by symmetry.
- Because extraction is differentiable, DMTet trains on losses measured on the final mesh, such as Chamfer distance and an adversarial loss.
- Selective volume subdivision makes cost grow with surface area rather than volume as resolution increases.
- NVIDIA released DMTet's core operations in its open-source Kaolin library, and GET3D uses DMTet to extract meshes.
Diepe duik
DMTet is a way of describing a 3D shape so that a neural network can both learn it easily and output it as a usable mesh. A mesh is the list of triangles that games, film tools and 3D printers expect. Neural networks find meshes hard to produce directly because the number of triangles and the shape's topology, meaning how many holes and separate pieces it has, must be fixed in advance. DMTet avoids that by learning a signed distance field (SDF), a function that gives every point in space its distance to the surface, negative inside the object and positive outside, and then extracting the mesh where that value crosses zero. Because the extraction step is differentiable, errors measured on the final mesh can be sent back to adjust the field. The paper calls this a hybrid of implicit and explicit representations.
The structure underneath is a tetrahedral grid. A tetrahedron is a pyramid with four triangular faces, and a grid of them can fill a cube completely, much like cubes fill a voxel grid. DMTet stores one SDF value at every corner, or vertex, of this grid, plus a small learned offset that lets each vertex move. The paper adopts this deformable grid from an earlier NVIDIA method called DefTet, but stores signed distances on vertices instead of occupancy on cells, which gives finer control over where the surface lies. Moving vertices lets the grid line up with thin parts such as chair legs or lamp stems without raising the overall resolution.
Turning the field into triangles uses marching tetrahedra, a cousin of the marching cubes algorithm. For each tetrahedron, the method checks the signs of its four corner values. Four signs give 16 combinations, which the paper notes reduce to three distinct cases by symmetry: no surface, one triangle or two triangles. Where an edge connects a negative corner to a positive one, the surface vertex is placed by linear interpolation at the point where the value would reach zero. Earlier work had argued that this formula blocks changes in topology during training because it divides by the difference of two values. The DMTet authors point out that the formula is only evaluated on edges whose corners have opposite signs, so the division never fails and gradients flow to both positions and SDF values.
The full network runs coarse to fine. An encoder called PVCNN turns the input, a point cloud or points sampled from a coarse voxel shape, into a 3D feature volume. A small fully connected network predicts an initial SDF value and feature for every grid vertex. The method then finds the surface tetrahedra, those whose corners disagree in sign, and a graph convolutional network predicts position offsets and SDF corrections for their vertices. Next comes volume subdivision: each surface tetrahedron and its neighbors are split into eight smaller ones by adding edge midpoints, and the rest of the grid is dropped. The paper notes that this keeps cost growing roughly with surface area, quadratically with resolution, rather than with volume. After extraction, a learnable version of Loop subdivision smooths and refines the mesh further.
Training compares the generated mesh to the target mesh. Points are sampled from both surfaces and the loss combines Chamfer distance, which measures how far each sampled point is from its nearest neighbor on the other surface, a normal consistency term, an adversarial loss from a 3D discriminator that judges local patches of the signed distance field, an SDF regularizer that keeps inside and outside consistent, and a penalty on large vertex movements. The authors report that DMTet beat ConvONet, an implicit method, and DECOR-GAN on their animal upsampling benchmark across all metrics, and that it ran about 10 times faster at inference than implicit-representation baselines. In their ablation, adding volume subdivision won 78% of human comparisons for overall look, and adding surface subdivision on top won 62%.
The obvious alternatives each lose something. Voxel grids are simple but their memory cost grows with the cube of resolution. Pure implicit networks, which predict an SDF or occupancy value for any queried point, handle any topology but are trained on sampled values rather than on the surface itself, and they need a separate and costly marching cubes pass to produce a mesh. Mesh-deformation methods that bend a template sphere give clean meshes but cannot add holes or separate parts. DMTet keeps arbitrary topology while supervising the final surface. In the paper's oracle test on ShapeNet chairs, marching tetrahedra captured thin structures better than marching cubes for the same number of queried points.
DMTet has known limits. Losses on the surface only reach vertices near the surface, so the paper needs an extra SDF loss to stop disconnected fragments from appearing. Flipping all signs in a tetrahedron yields the same triangles, so the surface alone cannot say which side is inside. Resolution is still bounded by the grid, and mesh quality depends on the number of subdivision levels. Later work such as FlexiCubes, published at SIGGRAPH 2023, argues that isosurface methods adapted from marching cubes or dual contouring lack the freedom to represent sharp, feature-preserving meshes during optimization or can be numerically unstable, and adds extra learnable parameters to address that.
In practice, DMTet needs a GPU, a tetrahedral grid file and a differentiable renderer or 3D loss. The paper's point cloud experiments report inference times on a single NVIDIA V100 GPU, and its base model uses a uniform grid of resolution 100. NVIDIA released the marching tetrahedra, volume subdivision and surface subdivision operations in its open-source Kaolin library, which is the usual starting point. Training data still matters: the animal results relied on 1,562 licensed models, and generators built on DMTet, such as GET3D, inherit the biases of their training collections.
04Uitgewerkt voorbeeld
Tracing marching tetrahedra on one cell
Take one tetrahedron with corners A, B, C and D and SDF values -0.2, 0.3, 0.4 and 0.1.
Mark the signs: A is inside (negative) and B, C and D are outside (positive), so this is the one-corner case and produces one triangle.
List the edges that change sign: A to B, A to C and A to D.
On edge A to B, place the vertex at fraction 0.2 / (0.2 + 0.3) = 0.4 of the way from A to B.
On edge A to C the fraction is 0.2 / 0.6, about 0.33, and on edge A to D it is 0.2 / 0.3, about 0.67.
Connect the three points to form one triangle, oriented so its normal points from A toward the positive corners.
Change A's value to -0.25 and recompute: every vertex slides slightly toward B, C and D, which is the gradient a training loss would use.
Wat het laat zien
The learner ends up with one surface triangle and an intuition for how changing an SDF value moves the mesh smoothly. It can be checked with Kaolin's marching_tetrahedra function on a single-tetrahedron grid using the same values.
Implementatie in de echte wereld
The original DMTet paper upsamples coarse 16 x 16 x 16 voxel animals, similar to blocky game models, into detailed meshes, training on 1,562 licensed TurboSquid animal models with 1,120 used for training and 442 for testing.
NVIDIA's GET3D generator, presented at NeurIPS 2022, learns from 2D image collections and uses DMTet to extract a surface mesh from its predicted signed distance field, producing textured cars, chairs, animals and characters.
NVIDIA's nvdiffrec method uses differentiable marching tetrahedra to recover triangle meshes, materials and lighting from multi-view photos, producing assets that render at interactive rates in standard triangle-based engines.
NVIDIA's Kaolin library ships a tutorial, Reconstructing Point Cloud with DMTet, that trains a signed distance estimator to rebuild a mesh from a point cloud using a Chamfer distance loss and marching tetrahedra.
In the paper's ShapeNet benchmark, DMTet reconstructs meshes from noisy point clouds of 5,000 points across 13 object categories, including chairs, lamps and airplanes.
The Future of DMTet: Deep Marching Tetrahedra for 3D Shapes
DMTet's main legacy is as a component. NVIDIA's GET3D (NeurIPS 2022) uses it to turn predicted signed distance fields into textured meshes trained only from 2D images, and nvdiffrec, first posted in November 2021, uses differentiable marching tetrahedra to recover meshes, materials and lighting from photos. These show a pattern in which a differentiable mesh extractor sits between a learned field and a rasterizer, so that 2D image losses can shape 3D geometry. Open problems center on mesh quality and resolution. FlexiCubes (SIGGRAPH 2023) argues that extraction layers adapted from marching cubes or dual contouring lack the degrees of freedom to keep sharp features and can be unstable during optimization, and proposes extra learnable parameters on top of a dual marching cubes grid. Researchers continue to compare tetrahedral and cubic grids for thin structures, memory use and triangle quality. Another direction is scale. DMTet was evaluated on curated sets, about 1,500 licensed animal models and 13 ShapeNet categories, and its quality outside such domains depends on the data. How well grid-based extractors handle open surfaces, very thin sheets and scenes rather than single objects remains an active question.
Het in de praktijk brengen
Waarom het ertoe doet
- Snelheid en schaal. Visuele AI kan inspectie-, detectie- en taggingtaken op schaal automatiseren.
- Bouwkeuzes. Creatieve teams kunnen concepten sneller prototypen met minder handmatige revisies.
- Team en workflow. Bij bewerkingen kan gebruik worden gemaakt van beeld- en videosignalen die voorheen moeilijk te verwerken waren.
Pas op voor
- Beeldrechten en toestemming kunnen juridische risico's worden als de herkomst onduidelijk is.
- De prestaties van modellen kunnen variëren afhankelijk van de belichting, demografische gegevens en omgevingen.
- Valse positieve resultaten kunnen onopgemerkt blijven, tenzij de vertrouwensdrempels worden gecontroleerd.
Volgende stappen
- Definieer acceptatiecriteria voor precisie-, terugroep- en foutkosten.
- Test met gegevens die overeenkomen met echte productieomstandigheden.
- Voeg menselijke beoordeling toe voor voorspellingen met weinig vertrouwen of hoge impact.
- Volg modelafwijkingen en valideer opnieuw na wijzigingen in de camera of dataset.
Bronnen en verdere literatuur
- arXivDeep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis
- NeurIPS 2021 ProceedingsDeep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis (paper PDF)
- NVIDIA Toronto AI LabDMTet project page
- NVIDIA Kaolin documentationTutorial Index (Reconstructing Point Cloud with DMTet)
- NVIDIA Toronto AI LabGET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
- arXivExtracting Triangular 3D Models, Materials, and Lighting From Images
- arXiv (SIGGRAPH 2023)Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
Blijf verkennen
Gratis nieuwsbrief
Ontvang de dagelijkse AI-briefing
Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.
Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.
Test jezelf
Take the DMTet: Deep Marching Tetrahedra for 3D Shapes quiz
Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.
Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie
Veelgestelde vragen
What is DMTet: Deep Marching Tetrahedra for 3D Shapes?
DMTet, short for Deep Marching Tetrahedra, is a 3D shape representation introduced by NVIDIA researchers at NeurIPS 2021. It stores a signed distance field on a deformable grid of tetrahedra and turns it into a triangle mesh with a differentiable step, so a neural network can be trained directly on the final surface. It became a standard building block for generating and reconstructing 3D meshes.
Welke twee ingrediënten combineert DMTet in zijn hybride representatie?
DMTet combineert een vervormbaar tetraëdrisch raster met voorspelde afstandswaarden om differentieerbare mesh-extractie mogelijk te maken.
Wat geeft de ondertekende afstandswaarde bij elk hoekpunt aan?
Een ondertekend afstandsveld codeert de afstand tot het oppervlak met een teken dat binnen versus buiten aangeeft, dus de nuldoorgang definieert het oppervlak.
Hoe wordt het expliciete driehoeksnetwerk geëxtraheerd in DMTet?
Een differentieerbare Marching Tetrahedra-stap plaatst oppervlaktehoekpunten overal waar de SDF van teken verandert over een tetraëderrand.
Wat maakt de oppervlakte-extractie van DMTet differentieerbaar ondanks veranderingen in de topologie?
Hoewel de topologie discreet verandert, planten gradiënten zich voort door de continue SDF-waarden en geleerde hoekpuntvervormingen die de locaties van oppervlaktepuntpunten bepalen.
Hoe voegt DMTet op efficiënte wijze fijne geometrische details toe?
Het verdeelt alleen de tetraëders dicht bij het oppervlak, waarbij de capaciteit wordt geconcentreerd waar details nodig zijn in plaats van op lege ruimte.
Blijf leren
Gerelateerde gidsen
Er zijn meer handleidingen voor dit onderwerp geselecteerd