I am currently building a text-to-image pipeline and I want to improve the alignment between my prompts and the generated images. I’ve read that Classifier-Free Guidance (CFG) is the industry standard, but I am confused about the training phase. Do I need to train two separate models, or can I use a single model with a "null" token? Also, how does the guidance scale actually modify the predicted noise during the reverse diffusion process? I’m worried about the computational cost of running two forward passes per step.
3 answers
To implement CFG effectively, you don't actually need two separate models. Instead, you train a single conditional model but randomly replace the conditioning input (like your text embedding) with a fixed "null" or "empty" token—usually about 10-20% of the time. This teaches the model to handle both conditional and unconditional generation simultaneously. During inference, you perform two forward passes: one with your prompt ($\epsilon_{pos}$) and one with the null token ($\epsilon_{neg}$). You then combine them using the formula: $\epsilon_{final} = \epsilon_{neg} + w \cdot (\epsilon_{pos} - \epsilon_{neg})$, where $w$ is your guidance scale. This "pushes" the image away from the generic unconditioned output and closer to your specific prompt requirements, significantly increasing fidelity.
That’s a clear explanation of the inference logic, Jennifer. However, I’ve noticed that when I crank my guidance scale above 10.0, the images start to look "burnt" or overly saturated with weird artifacts. Is there a mathematical reason why high CFG scales cause this dynamic range issue, and can it be fixed without lowering the scale?
In my production environment, the 2x inference cost was a dealbreaker. We eventually switched to "Guidance Cloning" or distilling the CFG process into a single-pass student model to keep the speed high.
I agree with Mark. For real-time apps, that extra forward pass is brutal. I'd also suggest looking into "Self-Guidance" techniques if you're working with very limited VRAM, as it can sometimes mimic the effects of CFG without the full second pass.
You're seeing "oversaturation" because the guidance formula effectively amplifies the magnitude of the noise vectors beyond what the model saw during training. To fix this, many people use "Dynamic Thresholding" or "Rescale CFG." Essentially, you calculate the standard deviation of the guided noise and rescale it back to match the original conditional noise's scale. In my experiments with Stable Diffusion, this allowed me to use a CFG of 20.0 for hyper-realistic textures without the "burnt" look you're describing.