I’ve been reading about DreamFusion and how it uses Score Distillation Sampling (SDS) to bypass the lack of massive 3D datasets. Can someone explain the mechanics of how it leverages 2D diffusion priors to optimize a 3D representation like NeRF? I'm curious about how it handles the trade-off between detail and the "Janus problem" where models end up with multiple faces.
3 answers
Score Distillation Sampling (SDS) is a game-changer because it allows us to optimize 3D assets without needing a direct 3D dataset, which is notoriously hard to curate. By treating a 3D representation like a Neural Radiance Field (NeRF) as a differentiable generator, SDS uses a pre-trained 2D diffusion model as a "critic." It renders the 3D model from a random camera angle, adds noise, and then calculates the gradient of the diffusion model's loss to push the 3D structure toward a more realistic 2D image. This iterative process essentially "distills" the vast knowledge of 2D image models into a consistent 3D form, allowing for incredible texture and shape detail that wasn't possible with previous voxel-based methods.
That explanation makes sense for the texture, but doesn't the gradient descent in SDS often lead to over-saturation or the "average" look? How do newer implementations specifically tackle the over-smoothing issues inherent in the original SDS loss function to ensure we get sharper edges and more realistic lighting in the final 3D mesh outputs?
SDS essentially acts as a bridge. It uses 2D image consistency to supervise 3D geometry. By backpropagating the score function, the model learns to align the 3D density with text prompts via the diffusion prior.
I agree with Jessica. The ability to use 2D priors is the only reason we are seeing such a boom in 3D assets for gaming and VR right now. Without SDS, we'd still be stuck with very basic geometric primitives.
Robert, that’s a sharp observation. Newer frameworks often swap SDS for Variational Score Distillation (VSD). While SDS treats the 3D parameters as a single point, VSD treats them as a distribution. This prevents the "over-smoothing" by modeling the score of the current rendered distribution rather than just using the 2D prior's score. It results in much higher visual diversity and sharper high-frequency details.