Same deterministic call paid for on every request

The same deterministic call paid for again on every duplicate request.

Rule result-cache-candidateCost audit

Why it costs money

Some calls return the same result for the same input. An embedding of a given text with a given model does not change. When an endpoint calls the provider every time, a retry, a page refresh or two users searching for the same phrase each pay full price for a result you already bought.

Embeddings are cheap per call, so this matters most at volume. Image generation from a fixed set of options costs far more per call, and repeats there add up quickly. A related case is maintenance code that reads text back out of storage, embeds it again unchanged and writes the new vector next to the one it already had.

Example

import OpenAI from "openai";
import { searchVectors } from "@/lib/vector-store";

const openai = new OpenAI();

export async function GET(req: Request) {
  const q = new URL(req.url).searchParams.get("q")?.trim().toLowerCase() ?? "";
  const res = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: q,
  });
  const hits = await searchVectors(res.data[0].embedding, 10);
  return Response.json(hits);
}

import { createHash } from "node:crypto";
import OpenAI from "openai";
import { Redis } from "@upstash/redis";
import { searchVectors } from "@/lib/vector-store";

const openai = new OpenAI();
const redis = Redis.fromEnv();
const MODEL = "text-embedding-3-small";

async function embed(text: string): Promise<number[]> {
  const cacheKey = "emb:" + MODEL + ":" + createHash("sha256").update(text).digest("hex");
  const cached = await redis.get<number[]>(cacheKey);
  if (cached) return cached;
  const res = await openai.embeddings.create({ model: MODEL, input: text });
  await redis.set(cacheKey, res.data[0].embedding, { ex: 60 * 60 * 24 * 30 });
  return res.data[0].embedding;
}

export async function GET(req: Request) {
  const q = new URL(req.url).searchParams.get("q")?.trim().toLowerCase() ?? "";
  return Response.json(await searchVectors(await embed(q), 10));
}

How to fix it

Build a cache key from everything that determines the output (the model name plus a hash of the input, or the full set of generation options), read the cache before calling the provider, and store the result with a TTL. A Redis or key-value store you already use is usually enough.

For re-embedding jobs, compare a hash of the stored text with the current text and skip the embedding call when they match. Only re-embed everything when you change the embedding model.

What the audit detects

• Embedding or image generation calls inside a request handler, in a file with no cache read of any kind (cache.get, redis.get, memoize, lru_cache, a cache key, hashing the inputs). Example code is skipped.
• Round-trip re-embeds: text read from a store, row or payload, embedded, and written straight back with update, upsert or similar, in a file with no cache.
• Chat completions are deliberately excluded: their output varies and is often specific to one user, so caching them is usually the wrong advice. Caching done by a gateway or CDN outside the file is not visible to it.

Research

GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings. Caching model responses keyed on the input removes redundant paid calls, and repeated or near-identical requests make up a meaningful share of production traffic.