I understand how text models work, but Generative AI for images (like Midjourney or Stable Diffusion) confuses me. What is the "Forward Diffusion" and "Reverse Diffusion" process? How does the model "learn" to create a cat from a cloud of random pixels, and what role does the text prompt play in guiding this process?
3 answers
It’s a process of "Learn to Destroy, then Learn to Reconstruct." In Forward Diffusion, the model takes an image of a cat and gradually adds Gaussian noise until it’s unrecognizable. In Reverse Diffusion, the model is trained to "predict" the noise that was added at each step and subtract it. By starting with pure random noise and following the prompt's guidance, it "cleans" the noise until a structured image emerges.
The text prompt acts as a "Conditioning" signal. As the model removes noise, it uses a Transformer-based text encoder (like CLIP) to ensure that the patterns it is "finding" in the noise align with your words. If you type "Space Marine," the model prioritizes finding metallic and galactic patterns during the de-noising steps.
Most modern models use Latent Diffusion. Instead of working on the actual pixels (which is computationally heavy), they work in a compressed "Latent Space." This makes image generation much faster and allows it to run on consumer-grade GPUs.
This is the genius of Stable Diffusion. By operating in latent space, it democratized high-quality image generation for everyone with a decent graphics card.
This is why "Negative Prompts" work. You are telling the model which patterns not to find in the noise as it reconstructs the image.