As our team grows, we're finding that small tweaks to the system prompt break features for other developers. We’re currently just saving prompts in .env files, which is getting messy. How do you version-control prompts like code? Are tools like LangSmith or Portkey worth it?
3 answers
Treating prompts like code is the right mindset. We moved away from .env files and started using "Prompt Management Systems." We use a Git-based approach where every prompt is a YAML file in its own repo. When we want to update a prompt, we create a PR and run an "A/B Eval" script that compares the old prompt's output to the new one using a set of 50 test cases. Tools like LangSmith are excellent for this because they provide a "Playground" where non-technical stakeholders can edit prompts and see the results before the engineers commit the changes to production.
This sounds great for development, but how do you handle "Model Drift"? If OpenAI updates the underlying model, your versioned prompt might suddenly behave differently.
We use a simple JSON file in our main repo. It’s version-controlled with the rest of the code, so a prompt change is always tied to a specific deployment tag.
JSON versioning is a good start! It ensures that if you need to rollback your application, the prompt rolls back with it, preventing those "logic mismatches."
Ronald, we handle this by pinning our model versions (e.g., gpt-4o-2024-05-13) instead of using the generic latest tag. This gives us a stable baseline. Whenever a new model is released, we run our entire "Evaluation Suite" against the new model version. We only "promote" the new model + prompt combo to production once we verify the pass rate is equal to or better than the previous one. Automated evals are the only way to sleep soundly when you're building on top of a third-party API that changes constantly.