Home / Rank #2
Prompt caching
Prompt caching reuses the computed state behind a repeated prompt prefix (system prompt, tool definitions, long documents), so the static part of every request bills at a steep discount — up to 90% off on Anthropic with explicit cache breakpoints, and ~50% automatically on OpenAI.
Production reports commonly land in the 50–80% cache-hit range once prompts are structured with the stable content first and the variable content last. For chat apps, agents, and anything with a large system prompt, this is the highest-leverage one-day fix available.
How to do it
- Restructure prompts: stable content (system prompt, tools, reference docs) first, variable content (user message) last.
- On Anthropic, add cache_control breakpoints to the stable blocks; on OpenAI, caching applies automatically to repeated prefixes over 1024 tokens.
- Keep the prefix byte-identical across requests — any change above the breakpoint invalidates the cache.
- Track your cache-hit rate; below ~50% usually means something volatile (timestamps, request IDs) is leaking into the prefix.
Frequently asked questions
Does caching change the model output?
No. Prompt caching reuses computation for identical input prefixes; outputs are unaffected. It is purely a billing and latency optimization.
How long do caches live?
Provider-dependent: Anthropic offers 5-minute and 1-hour TTLs, OpenAI caches typically persist minutes and extend while in use. High-traffic prompts stay hot on their own.
Tools for this method
LiteLLM
The default self-hosted gateway: one OpenAI-compatible proxy across 100+ providers with budgets, caching, routing, fallbacks, and …
Portkey
Managed gateway with the strongest built-in semantic caching, plus guardrails, routing, and cost analytics. The low-ops route to t…
Next method: #3 Model routing & cascades