DreamFusion und Score Distillation Sampling

DreamFusion turns a sentence such as 'a squirrel playing a guitar' into a 3D model that can be viewed from any angle, relit and placed in a scene. It was introduced in 'DreamFusion: Text-to-3D using 2D Diffusion' by Ben Poole, Ajay Jain, Jonathan T. Barron and Ben Mildenhall of Google Research and UC Berkeley, posted to arXiv on 29 September 2022. The paper won an Outstanding Paper Award at ICLR 2023. Its central contribution is a loss function called Score Distillation Sampling, or SDS, which lets a pretrained image generator supervise the creation of something it was never trained to make.

The problem it solved was data. Text-to-image diffusion models learned from billions of captioned images, but no comparable collection of captioned 3D models existed, and the authors note that efficient architectures for denoising 3D data were also lacking. Earlier text-to-3D work such as Dream Fields used CLIP, a model that scores how well an image matches a caption, to guide a 3D scene, but its results tended to lack realism. DreamFusion kept the same overall recipe, optimizing a 3D scene so its 2D renderings look right, and swapped CLIP for a diffusion model.

The 3D scene is a Neural Radiance Field, or NeRF: a small neural network that takes a 3D point and returns a density (how opaque the space is) and a color. Images are made by volume rendering, shooting rays from a virtual camera and accumulating density and color along each ray. DreamFusion builds on mip-NeRF 360, a variant that reduces aliasing. Rather than raw color, it predicts albedo, the surface's base color, and computes surface normals from the density field so the object can be shaded under a light source. This matters because shading exposes the true shape.

Each optimization step runs as follows. Sample a random camera position around the object and a random light direction. Render an image of the current NeRF, sometimes with full color and shading, sometimes as plain albedo, and sometimes as a 'textureless' gray shaded render that shows geometry alone. Add Gaussian noise to that image at a random noise level. Ask the frozen diffusion model, given the noisy image and the prompt, to predict the noise. Take the difference between predicted and actual noise as the update direction, and backpropagate it through the renderer into the NeRF's weights. The diffusion model is never updated and gradients never pass through it, so it behaves like a fixed critic suggesting image-space edits. Over 15,000 iterations, renderings from every angle converge toward images the diffusion model considers likely for the prompt.

Two practical tricks made it work. First, prompts are adjusted by camera angle: 'overhead view' for high elevations, and blends of 'front view', 'side view' or 'back view' by azimuth. Without this, the paper shows a dog with multiple faces, because the image model draws a front-facing dog from every direction, a failure later called the Janus problem. Second, classifier-free guidance, which strengthens the prompt's influence, was set to 100, far higher than the single-digit values typical for image generation, because lower values produced oversmoothed results under SDS. The diffusion model was Google's Imagen, using only its 64 by 64 pixel base model. Each scene took around 1.5 hours on a TPUv4 machine with four chips.

On quantitative evaluation, the paper used CLIP R-Precision, the accuracy with which CLIP picks the correct caption for a rendering among distractors. With the CLIP L/14 model, DreamFusion scored 79.7 percent on color renders and 58.5 percent on textureless geometry renders, the latter showing that the shape itself matched the prompt. The authors note that Dream Fields and CLIP-Mesh had an advantage on this metric because they trained with CLIP.

Follow-up work moved fast. NVIDIA's Magic3D, posted in November 2022, added a second stage that refines a textured mesh with a high-resolution latent diffusion model, reporting about 40 minutes per object, twice as fast as DreamFusion, and a 61.7 percent rater preference over it. ProlificDreamer, posted in May 2023, generalized SDS into variational score distillation, treating the 3D scene as a distribution rather than a single point, and reported better quality and diversity at a normal guidance weight of 7.5. Open-source frameworks such as threestudio collected these methods; its documentation notes that DreamFusion guided by DeepFloyd IF performs much better than the Stable Diffusion variant.

The limitations are well documented by the authors. SDS tends to produce oversaturated and oversmoothed results compared with ordinary diffusion sampling, and outputs vary little across random seeds, which the paper attributes to the mode-seeking nature of its reverse KL divergence objective. The 64-pixel guidance model limits fine detail. Recovering 3D from 2D views is ill-posed, since many 3D shapes produce the same images, so the optimization can fall into poor local minima. The method also needs minutes to hours of optimization per object, unlike feed-forward 3D generators that output a model in one pass. In practice, the threestudio implementation lists about 6 GB of GPU memory for training the Stable Diffusion version and about 10 GB for the DeepFloyd IF version, plus about 15 GB to extract text embeddings.