I’m struggling with my LLM intermittently outputting markdown when I specifically asked for JSON. Can I use Guardrails AI to strictly enforce a schema? I want the guard to automatically re-prompt the model if the formatting is incorrect so my downstream API doesn't crash on a parsing error.
3 answers
This is actually one of the core strengths of the framework. By using a RAIL spec (which is XML-based), you can define the exact structure of the JSON you expect, including types like 'string', 'integer', and even nested objects. When the LLM provides an output, Guardrails AI attempts to parse it against your schema. If it fails, the 're-ask' mechanism takes over. It sends the error message and the original prompt back to the model, telling it exactly where the JSON was malformed. This usually fixes the issue in a single retry. It’s much more reliable than just relying on system prompts alone, which LLMs often ignore when they get creative.
Does the re-prompting significantly increase your token usage? I'm worried about the cost of multiple retries for complex schemas.
You should also check the pydantic integration. It lets you define your guards using standard Python classes, which is way more intuitive for most developers.
Diane is right! Pydantic makes the schema definition part so much cleaner compared to the old XML RAIL files.
It does add to the cost, but it's usually cheaper than a manual support ticket when an API call fails. In my tests, about 95% of formatting errors are fixed in the first retry. For very complex schemas, I recommend using the 'Fix' strategy where the guard tries to programmatically close missing brackets or quotes before resorting to a full re-prompt to the LLM.