I am attempting to train a new Textual Inversion (TI) embedding to capture a specific architectural style for my design projects. I’m using a dataset of 20 high-quality 512x512 images, but I’m torn on the "vectors per token" setting. I’ve seen some guides suggest 1-2 for better editability, while others go up to 16 for better likeness. What is the actual impact on the prompt's token limit, and does increasing the vectors lead to faster overfitting or "concept bleeding" into the rest of the generated scene?
3 answers
The number of vectors per token essentially determines the "vocabulary space" your concept occupies. In Stable Diffusion, you have a 75-token limit; if you use 16 vectors for one TI, you only have 59 tokens left for the rest of your prompt. From my experience, 4 to 8 vectors is the sweet spot for complex styles or specific objects. If you go too high, like 16 or 32, the embedding becomes too "heavy" and starts to ignore the rest of your prompt instructions, essentially forcing the training images' backgrounds into every generation. For a simple architectural style, I would recommend starting with 2-4 vectors. This preserves the model's ability to change the lighting or weather in the scene while still keeping the structural integrity of your style.
That’s a great breakdown of the token budget, Emily. But have you noticed a difference in how the learning rate should scale with more vectors? If I use 8 vectors instead of 2, should I lower the learning rate to prevent the embedding from collapsing into "NaN" errors during the first few hundred steps?
I always tell people to focus more on their "captioning" than the vector count. If you use BLIP to describe the backgrounds of your training images, the TI learns to ignore those parts and only focuses on the style.
I agree with Jessica. Proper preprocessing is 90% of the battle. If you don't caption out the "blue sky" in your training images, your architectural TI will always force a blue sky into every generation. I'd also suggest using "EasyNegative" as a backup to keep the outputs clean while testing your new embedding.
That is a very valid concern, Emily. When you increase the vector count, you’re essentially increasing the degrees of freedom the optimizer has to manage. I typically keep my learning rate at $5 \times 10^{-3}$ for 1-2 vectors, but drop it to $1 \times 10^{-3}$ if I’m pushing 8 or more. This prevents the "explosion" of weights. Also, make sure your initialization text is highly relevant—if you're training a style, use "style" or "art" as the init word so the vectors start in a mathematically useful region of the latent space.