DMTet: Deep Marching Tetrahedra for 3D Shapes

DMTet is a way of describing a 3D shape so that a neural network can both learn it easily and output it as a usable mesh. A mesh is the list of triangles that games, film tools and 3D printers expect. Neural networks find meshes hard to produce directly because the number of triangles and the shape's topology, meaning how many holes and separate pieces it has, must be fixed in advance. DMTet avoids that by learning a signed distance field (SDF), a function that gives every point in space its distance to the surface, negative inside the object and positive outside, and then extracting the mesh where that value crosses zero. Because the extraction step is differentiable, errors measured on the final mesh can be sent back to adjust the field. The paper calls this a hybrid of implicit and explicit representations.

The structure underneath is a tetrahedral grid. A tetrahedron is a pyramid with four triangular faces, and a grid of them can fill a cube completely, much like cubes fill a voxel grid. DMTet stores one SDF value at every corner, or vertex, of this grid, plus a small learned offset that lets each vertex move. The paper adopts this deformable grid from an earlier NVIDIA method called DefTet, but stores signed distances on vertices instead of occupancy on cells, which gives finer control over where the surface lies. Moving vertices lets the grid line up with thin parts such as chair legs or lamp stems without raising the overall resolution.

Turning the field into triangles uses marching tetrahedra, a cousin of the marching cubes algorithm. For each tetrahedron, the method checks the signs of its four corner values. Four signs give 16 combinations, which the paper notes reduce to three distinct cases by symmetry: no surface, one triangle or two triangles. Where an edge connects a negative corner to a positive one, the surface vertex is placed by linear interpolation at the point where the value would reach zero. Earlier work had argued that this formula blocks changes in topology during training because it divides by the difference of two values. The DMTet authors point out that the formula is only evaluated on edges whose corners have opposite signs, so the division never fails and gradients flow to both positions and SDF values.

The full network runs coarse to fine. An encoder called PVCNN turns the input, a point cloud or points sampled from a coarse voxel shape, into a 3D feature volume. A small fully connected network predicts an initial SDF value and feature for every grid vertex. The method then finds the surface tetrahedra, those whose corners disagree in sign, and a graph convolutional network predicts position offsets and SDF corrections for their vertices. Next comes volume subdivision: each surface tetrahedron and its neighbors are split into eight smaller ones by adding edge midpoints, and the rest of the grid is dropped. The paper notes that this keeps cost growing roughly with surface area, quadratically with resolution, rather than with volume. After extraction, a learnable version of Loop subdivision smooths and refines the mesh further.

Training compares the generated mesh to the target mesh. Points are sampled from both surfaces and the loss combines Chamfer distance, which measures how far each sampled point is from its nearest neighbor on the other surface, a normal consistency term, an adversarial loss from a 3D discriminator that judges local patches of the signed distance field, an SDF regularizer that keeps inside and outside consistent, and a penalty on large vertex movements. The authors report that DMTet beat ConvONet, an implicit method, and DECOR-GAN on their animal upsampling benchmark across all metrics, and that it ran about 10 times faster at inference than implicit-representation baselines. In their ablation, adding volume subdivision won 78% of human comparisons for overall look, and adding surface subdivision on top won 62%.

The obvious alternatives each lose something. Voxel grids are simple but their memory cost grows with the cube of resolution. Pure implicit networks, which predict an SDF or occupancy value for any queried point, handle any topology but are trained on sampled values rather than on the surface itself, and they need a separate and costly marching cubes pass to produce a mesh. Mesh-deformation methods that bend a template sphere give clean meshes but cannot add holes or separate parts. DMTet keeps arbitrary topology while supervising the final surface. In the paper's oracle test on ShapeNet chairs, marching tetrahedra captured thin structures better than marching cubes for the same number of queried points.

DMTet has known limits. Losses on the surface only reach vertices near the surface, so the paper needs an extra SDF loss to stop disconnected fragments from appearing. Flipping all signs in a tetrahedron yields the same triangles, so the surface alone cannot say which side is inside. Resolution is still bounded by the grid, and mesh quality depends on the number of subdivision levels. Later work such as FlexiCubes, published at SIGGRAPH 2023, argues that isosurface methods adapted from marching cubes or dual contouring lack the freedom to represent sharp, feature-preserving meshes during optimization or can be numerically unstable, and adds extra learnable parameters to address that.

In practice, DMTet needs a GPU, a tetrahedral grid file and a differentiable renderer or 3D loss. The paper's point cloud experiments report inference times on a single NVIDIA V100 GPU, and its base model uses a uniform grid of resolution 100. NVIDIA released the marching tetrahedra, volume subdivision and surface subdivision operations in its open-source Kaolin library, which is the usual starting point. Training data still matters: the animal results relied on 1,562 licensed models, and generators built on DMTet, such as GET3D, inherit the biases of their training collections.