LLM cost monitoring vs static analysis: use both

Runtime LLM cost monitoring (provider dashboards, Langfuse, Helicone) and static analysis before deploy answer different questions. What each sees and misses.

Updated September 24, 20264 min read

LLM cost monitoring and static analysis answer different questions, so most teams running AI features in production benefit from both. Runtime monitoring (provider dashboards, tracing tools such as Langfuse, gateways such as Helicone) tells you what your application actually spent, on which requests, and when. Static analysis reads the code before it ships and tells you which patterns will make spend grow or run unbounded, including on code paths that have not run yet.

Put simply: monitoring measures the bill, static analysis explains the code that produces it. Monitoring needs traffic and instrumentation; static analysis needs neither but cannot see traffic at all.

What runtime monitoring gives you

Provider dashboards

Every major provider shows usage and cost in its console, typically broken down by model, API key or project, and by day. This is free, needs no code, and is the authoritative number, since it is what you are billed. Most providers also offer spend alerts or limits at the organisation or project level. What dashboards do not know is your application's structure: they cannot tell you which feature, endpoint, user or agent run a cost came from, unless you separate keys or projects along those lines.

Tracing and observability tools

Tools such as Langfuse (open source, self-hostable or hosted) record each model call as part of a trace, with tokens, cost, latency, inputs and outputs, and let you attach your own metadata such as user ID, feature or session. That makes cost attributable: this agent run cost $0.40, this user spent $12 this week, this prompt version doubled average tokens. Such tools also commonly support evaluations and prompt management. They require instrumentation, through an SDK, a framework integration or OpenTelemetry.

Gateways and proxies

Helicone and similar gateways sit between your application and the provider, often by changing the base URL. Because every request passes through them, they can log usage and cost with little code change, and gateways commonly add features such as caching and rate limiting at the network layer. The trade-off is another hop in the request path and another service to trust with prompts and keys.

What runtime monitoring cannot tell you

• Anything before deploy. A missing output cap or an unbounded loop shows up after it has cost money.
• Paths that rarely run. An error handler that falls back to an expensive model, a retry path, a monthly job. Until they run, there is nothing to see, and when they do run, the damage is often done in one burst.
• The cause in code. A trace shows a run with 200 steps. It does not show that max_steps defaults to None in a config class three files away.
• What was never instrumented. A call added last week outside the wrapped client, or a service nobody connected to the tracing backend, is invisible.
• Security exposure. A key in a browser bundle or an unauthenticated endpoint looks like normal traffic until someone finds it.

What static analysis gives you

A static audit reads the source and finds each place the code calls a model, then checks the surrounding code: is there an output cap, is the call inside a loop, is the loop bounded, is there a budget guard, is the endpoint authenticated, is the full history re-sent, is the cached prefix stable. It works on a branch before merge, needs no credentials and no traffic, and points at a file and line.

# A static check can flag this before it ever runs:
@app.post("/chat")                       # no auth dependency on the route
async def chat(body: ChatBody):
    history.append(body.message)
    while True:                          # no iteration bound
        res = client.messages.create(
            model=body.model,            # client picks the model
            messages=history,            # full history every turn
            max_tokens=4096,
        )
        ...

What static analysis cannot tell you

• Real volume. It does not know how many requests you serve, how long conversations get, or how many steps agents take. Any dollar figure it gives is an estimate built on stated assumptions.
• Whether a pattern matters at your scale. An unbounded loop in an admin tool used twice a month is not the same risk as one in a public endpoint.
• Controls outside the repository. A gateway rate limit, a provider-level spend cap or an infrastructure quota is invisible to a scan of application code.
• Dynamic behaviour. Models chosen from a database at runtime, prompts assembled across many functions, and calls made through unusual wrappers can be missed or partially understood.
• Everything. Static rules find patterns someone thought to write down. They miss problems nobody has encoded yet.

How they fit together

1. Before merge: run a static audit on the branch. Fix unbounded loops, missing caps, missing guards and exposed keys while the change is small.
2. At deploy: make sure every call goes through an instrumented client, and set provider-level spend alerts as a backstop.
3. In production: use tracing to attribute cost to features, users and runs, and alert on anomalies such as cost per run or tokens per request jumping after a release.
4. When monitoring shows a spike: use the trace to find the run, then read the code path (or re-run the static audit) to find the cause and the fix.
5. When the audit reports a pattern: use monitoring data to decide priority. A finding on a call site with high real volume comes first.

The two also check each other. If the audit lists call sites that never appear in traces, either they are dead code or they are not instrumented, and both are worth knowing. The patterns themselves are covered in How to reduce LLM API costs, with deeper guides on agent loops, spending limits and caching.

How IntentiAI helps

IntentiAI is the static half. npx intentiai finds every model call site in a repository (TypeScript, JavaScript, Python, Go, Java, Ruby, PHP) and checks it against 21 cost rules, such as agent-loop-unbounded, no-budget-guard, no-max-tokens and recurring-job-llm, the last being a pattern per-request tracing easily misses. It runs locally with no API key, uploads findings only (never source) to a private report link, and --local uploads nothing. --json gives machine-readable output for CI.

It does not replace a monitoring tool and does not see traffic. On the project's measurements it found 98.6% of model calls across seven hand-labelled open-source apps, and 45% of hand-labelled cost problems in five apps the rules were not written against, so a clean result is not a guarantee. Use it alongside whatever runtime monitoring you already have. See the cost audit and the docs.