TokelyTHE SHORT ANSWER
Prompt caching reuses the processing of repeated input while generating a new response. Semantic caching returns a stored answer when a new request is sufficiently similar to an earlier one. Prompt caching can lower input charges; semantic caching can avoid a generation call entirely, but it must also prove that the old answer is still appropriate.
What does prompt caching actually reuse?
A long system instruction, tool definition or document prefix may appear unchanged across many requests. A provider can reuse eligible processing for that prefix. The response itself is still generated and output tokens remain billable.
Keep stable content before variable content where the API’s caching design requires a shared prefix. A changing timestamp near the beginning can reduce reuse. Check minimum lengths, expiry, cache scope and usage fields for the specific route. OpenAI prompt caching.
When does a cache write pay for itself?
Use the formula: cached cost = write cost + repeated reads × read cost. Compare it with processing the same prefix from scratch on every request.
For a hypothetical provider, suppose a prefix costs $0.02 normally, $0.025 to write and $0.002 to read. Ten uses cost $0.20 without caching and $0.043 with one write plus nine reads. That saves $0.157 on the prefix, or 78.5%. It does not reduce the separate output bill by 78.5%.
These example prices are illustrative. Actual cache-write and read rates vary by model, lifetime and provider. Check the relevant Anthropic caching documentation when using its API format.
What makes semantic caching different?
A semantic cache compares a new query with stored queries, commonly using embeddings. If an eligible match is close enough, the application returns a stored answer. Similar wording alone is insufficient: “Can I cancel?” may have different answers for different customers, plans and dates.
A cache key should incorporate tenant, permissions, language, relevant product version and data freshness. Apply authorization filters before retrieving a cached answer. Do not search across customer boundaries and rely on a similarity threshold to protect private information.
How do you decide whether semantic caching is safe?
Start with stable public information such as documented product behavior. Avoid using it indiscriminately for balances, inventory, personalized recommendations or actions. A wrong cached answer may look fluent while referring to stale facts.
Measure false matches as well as hit rate. A high hit rate is not an achievement if it sends yesterday’s policy to today’s customer. Invalidate entries when source material changes and retain a way to bypass the cache.
What does Tokely support?
Tokely’s Messages API guide documents prompt-cache usage fields and separate cache billing. Do not assume it provides an application-level semantic answer cache. You can build that layer in your application using an embedding model and a store you control.
Check Tokely’s embedding guide for vector generation, and compare cache costs using the actual billing categories returned by your chosen route.