I'm interested in the "Semantic Cache" feature. How does Helicone determine if two prompts are "similar" enough to serve a cached response? Is it reliable for a customer support bot where the wording might vary but the intent is exactly the same?
3 answers
Semantic caching is a bit more advanced than standard exact-match caching. Helicone uses vector embeddings to compare the "meaning" of the incoming prompt with previously stored results. You can configure a "similarity threshold" (usually between 0.85 and 0.95). If the cosine similarity of the new prompt is higher than that threshold, it serves the cached result. For a support bot, this is incredibly effective for common questions like "How do I reset my password?" Even if the user says "I forgot my login," the semantic cache will recognize the intent and serve the answer instantly. This saves you the full cost of the LLM call and drops the latency from seconds to milliseconds.
What happens if the cache serves an answer that is slightly out of date? Is there a way to force a refresh or set an expiration time?
Just be careful with the threshold. If you set it too low, you might get "false positives" where the bot gives a slightly wrong answer.
That’s a valid concern, Logan. We found that 0.92 is the "Goldilocks" zone for our specific dataset to avoid those irrelevant responses.
You can set a Cache-Control header to define the TTL (Time to Live) for your responses. If your support docs change, you can also clear the cache via the API or the dashboard. We usually set a 24-hour TTL for our dynamic content, which ensures users get relatively fresh information while still reaping the performance benefits of the cache during peak hours.