We are running an experiment with three different system prompts. Is there a way to use Helicone to tag these requests so we can compare the average latency and cost of each version in the dashboard? I want to avoid building a custom logging database for our experiments.
3 answers
Custom properties are the perfect tool for this. You can add a header like Helicone-Property-Experiment-Group: Version-A to your requests. In the dashboard, you can then group your analytics by this specific property. This will show you a side-by-side comparison of the cost, request count, and even the "Time to First Token" for each prompt variant. It’s incredibly helpful for seeing if a longer, more detailed prompt is actually providing enough value to justify the extra cost and wait time. You can even filter your traces to see where one version might be failing more frequently than the others without needing any external data visualization tools.
Can we update the properties after the request has been made? Sometimes we only know if a result was "good" after the user interacts with it later.
You should use the "Request Tags" feature if you have more than 5 variables. It keeps the dashboard a bit cleaner than just using properties.
That’s a good tip, Frank. I’ll look into the tagging documentation to see how it differs from the standard header-based properties we are using.
Unfortunately, headers are fixed at the time of the request. However, you can use the Feedback API to send a score or a tag back to that specific request ID later. It’s not exactly a "property update," but it achieves the same goal of enriching the data post-inference so you can filter for "Successful" vs "Failed" experiments in your final report.