IntentiAI

Prompt caching vs response caching for LLM APIs

Provider prompt caching discounts repeated prefixes; response caching skips the call. How each works, when churn and storage fees make caching cost more.

4 min read

Prompt caching and response caching solve different problems. Provider prompt caching makes a call cheaper by reusing work on a prompt prefix the provider has already seen; you still make the call and still pay for output. Response caching skips the call entirely by storing the result in your own cache and returning it for a duplicate request. Prompt caching helps when many calls share a long, identical beginning. Response caching helps when many calls are identical and the answer does not need to change.

Both can go wrong in ways that raise cost instead of lowering it. Prompt caching on content that changes every call pays a write premium and never gets a read. Some explicit caches bill storage by the hour until deleted. Response caching on the wrong kind of call returns stale or wrong answers.

How provider prompt caching works

Providers cache the processed form of a prompt prefix. A later request that starts with exactly the same tokens reuses it, and the cached portion is billed at a discount. Details differ by provider and change over time, so check current documentation, but at the time of writing the broad shapes are:

  • OpenAI caching is on by default for supported models once a prompt reaches the model's minimum cacheable length (1,024 tokens on current models), and cached input tokens are billed at a steep discount (0.1x the uncached rate on recent models). You benefit if your prompts share a long static prefix.
  • Anthropic caches where you mark it with cache_control breakpoints. Per Anthropic's documentation, a cache write costs 1.25x the base input price for the default 5-minute lifetime (2x for the 1-hour option), and a cache read costs 0.1x. Each read refreshes the lifetime.
  • Google Gemini offers implicit caching on some models and explicit context caches that you create, reference by name, and delete. Explicit caches are billed for storage per token per hour for as long as they exist.

The rule that matters for all of them: the cache matches a prefix, byte for byte. Anything that varies, placed before the static content, breaks the match for everything after it.

Cache write churn: paying to write and never reading

With explicit breakpoints, a write is more expensive than a normal input token. That premium pays off only if later requests read the same prefix. If the cached block contains something that changes every call, such as a timestamp, a request ID, the user's name or the latest retrieved documents, every request writes a new cache entry and none reads one.

Example: 10,000-token system prompt, 1,000 calls within the cache lifetime.
Costs in base-input-token equivalents, using write = 1.25x and read = 0.1x.

  No caching:              1,000 * 10,000                  = 10,000,000
  Stable prefix, cached:   1 * 12,500 + 999 * 1,000        =  1,011,500   (about 90% less)
  Timestamp in the prefix: 1,000 * 12,500                  = 12,500,000   (25% more than no caching)
// Churns: the date and user name are inside the cached block.
system: [
  {
    type: "text",
    text: "Today is " + new Date().toISOString() + ". User: " + user.name + "\n" + LONG_INSTRUCTIONS,
    cache_control: { type: "ephemeral" },
  },
],

// Stable: the static instructions are cached; the varying part comes after the breakpoint.
system: [
  { type: "text", text: LONG_INSTRUCTIONS, cache_control: { type: "ephemeral" } },
  { type: "text", text: "Today is " + today + ". User: " + user.name },
],

Order prompts from most stable to least stable: tool definitions, system instructions, long reference documents, then conversation history, then the new user message. Watch the usage fields each response returns (for Anthropic, cache_creation_input_tokens and cache_read_input_tokens; for OpenAI, cached tokens in the usage details). If creation stays high and reads stay near zero, the prefix is not stable. See cache-write-churn.

Cache storage cost: the cache nobody deleted

Explicit caches with storage billing behave like a running server: they cost money per hour whether or not anything reads them. Two patterns cause trouble. The first is a cache created and never deleted, so storage runs for the full lifetime even when the job finished minutes after creating it. The second is a cache created per request, which multiplies storage by request volume and gives up most of the discount because each cache is used once.

Illustrative arithmetic (storage rate of $1 per 1M tokens per hour, check current pricing):
  One 200,000-token cache kept for 24 hours:     0.2 * $1 * 24  = $4.80
  A 200,000-token cache per request, 1-hour TTL,
  1,000 requests a day:                          1,000 * 0.2 * $1 = $200 a day in storage alone
from google import genai
from google.genai import types

client = genai.Client()

cache = client.caches.create(
    model=MODEL,
    config=types.CreateCachedContentConfig(
        contents=[large_document],
        ttl="900s",  # explicit and short: long enough for this batch of questions
    ),
)
try:
    for question in questions:
        client.models.generate_content(
            model=MODEL,
            contents=question,
            config=types.GenerateContentConfig(cached_content=cache.name),
        )
finally:
    client.caches.delete(name=cache.name)

Create caches once per document or session, not per request, set an explicit short lifetime, and delete them in a finally block when the work is done. See cache-storage-leak.

Response caching: not calling at all

A response cache stores the output of a call keyed on everything that determines it. On a hit, no tokens are billed. This is the right tool for calls whose correct answer does not depend on who is asking or when: classifying a product description, extracting fields from a document, summarising a fixed article, embedding a piece of text, translating a UI string.

import { createHash } from "node:crypto";

async function cachedCompletion(params: CompletionParams): Promise<string> {
  // Key on everything that changes the output: model, messages, and sampling settings.
  const key = "llm:" + createHash("sha256").update(JSON.stringify(params)).digest("hex");
  const hit = await cache.get(key);
  if (hit !== null) return hit;

  const res = await openai.chat.completions.create(params);
  const text = res.choices[0]?.message.content ?? "";
  await cache.set(key, text, { ttlSeconds: 60 * 60 * 24 * 7 });
  return text;
}
  • Include the model name and prompt version in the key, so changing either invalidates old entries.
  • Set a TTL even for stable tasks, so a bad cached answer eventually ages out.
  • Do not cache personalised, time-sensitive or creative outputs, or anything the user expects to vary on retry.
  • For embeddings, store a hash of the text next to the vector and skip re-embedding when the text is unchanged. Re-embedding unchanged records on every update is a common silent cost.

Semantic caching extends this by returning a stored answer for a request that is similar, not identical, using embedding distance. GPTCache (Bang, 2023) is a well-known open-source example. It can raise hit rates a lot, but a similarity threshold that is too loose returns the answer to a different question, so treat it as a product decision and evaluate it on real traffic. See result-cache-candidate.

Which one to use

  • Long shared instructions or documents, varied questions: prompt caching, with the stable part first.
  • Identical requests with a fixed correct answer: response caching, keyed on a hash of the inputs.
  • Many questions about one large document for a short time: an explicit context cache with a short TTL and a delete.
  • Both apply often. A chat app can prompt-cache its system prompt and response-cache its title generation.

For the wider picture of what inflates an LLM bill, see How to reduce LLM API costs.

How IntentiAI helps

Three cost-audit rules cover caching. cache-write-churn flags cache_control on content that interpolates values likely to change per call. cache-storage-leak flags explicit caches that are created and never deleted, created per request, or created without an explicit lifetime. result-cache-candidate flags deterministic calls served from request handlers with no result cache, and stored text that is re-embedded unchanged. Run npx intentiai in the repository.

These rules are heuristics with medium confidence: the varying value may sit outside the cached block, or deletion may happen in a wrapper the scan did not see, and each finding states that risk. A static scan also cannot measure your hit rate. Confirm with the cache usage fields in real responses. More in the docs and on the cost audit page.

More guides

Prompt caching vs response caching for LLM APIs | IntentiAI