Sample reports

These are real outputs of the IntentiAI CLI: a cost audit of the open-source roomGPT app, and a Ship Check of a test repo where a change with 13 planted mistakes was left uncommitted. Nothing on this page was written by hand or edited afterwards.

Cost audit of roomGPT

From npx intentiai <repo> --local on Nutlope/roomGPT. Upload is metadata only: file and line, rule, and the estimate with its assumptions.

LLM cost audit of roomGPT

commit 611398c · TypeScript/JavaScript · scanned Sep 23, 2026, 7:27 PM UTC · rules v0.1

Call sites: 1
Cost findings: 3
Warnings: 2
Info: 0

Estimated exposure

No call site names a model literal, so no repo-wide figure is given. Per-finding estimates below state the reference model they assume.

Findings ranked by severity

Cost
A billable async job is polled to completion with no deadline and no cancel path — if the poller dies the job keeps billing with nobody reading the result.
app/generate/route.ts:69rule orphaned-async-jobconfidence medium
Estimate full job cost per abandoned run, ×2 when the user retries — every timeout is a double charge
Generation jobs are billed by the provider from the moment they are submitted, for the whole run. If this poll loop is killed (serverless timeout, client disconnect, restart) the job is never cancelled: you pay in full for a result nobody receives, and the user retries and pays again.
Fix: Bound the wait with a deadline, and on timeout or client abort call the provider's cancel endpoint before returning. Better: use the provider's wait/webhook mode so there is no poll loop to abandon.
Backed by Replicate / OpenAI / Azure async job APIs (primary provider docs) (replicate.com): Async generation jobs bill for the full run once submitted, independent of whether the result is ever fetched; providers expose an explicit cancel endpoint precisely so an abandoned job can be stopped.
Cost
No dollar/token budget guard or circuit breaker found anywhere; LLM spend is unbounded.
app/generate/route.ts:40rule no-budget-guardconfidence high
Estimate unbounded — no cap on the default path
A runaway loop, retry storm, or traffic spike keeps calling the API with nothing to halt spend. 1 LLM call site(s) are exposed.
Fix: Add a per-run token/dollar budget that halts LLM calls when exceeded.
Backed by When Agents Do Not Stop: Uncovering Infinite Agentic Loops in LLM Agents (arXiv:2607.01641): Cost exhaustion is a core agentic-loop harm; a spend guard is the backstop.
Cost
The spend/abuse guard is constructed only when its env vars are set, and the request path skips it when they are not — the default deployment is unguarded.
app/generate/route.ts:8rule guard-conditional-on-envconfidence medium
Estimate unbounded whenever the guard's env vars are unset — the default path has no ceiling
Whether anything limits spend depends on deployment configuration, not on this code. If the env vars are missing (a fresh deploy, a preview environment, a fork), every request reaches the paid call with no quota, which is a Denial-of-Wallet opening on a public endpoint.
Fix: Fail closed: in production, refuse to serve (503) when the limiter cannot be constructed, rather than silently skipping it. Add the guard's env vars to the documented deploy configuration so the default deployment is protected.
Backed by OWASP LLM10:2025 Unbounded Consumption / Denial of Wallet (genai.owasp.org): A per-user quota that is absent in the default configuration provides no protection: unbounded consumption is governed by the deployed configuration, not by the presence of guard code.
Warning
Request-body fields reach a paid generation call with no schema or allowlist validation in the handler.
app/generate/route.ts:40rule unvalidated-input-to-paid-callconfidence medium
Estimate one full billed generation per abusive request — volume bounded only by the attacker
Without a check on what the body may contain, the endpoint serves any workload a caller submits, not just your product's. Every abusive request is a full billed generation, and the volume is set by the attacker rather than by your users.
Fix: Validate the body before the paid call: check each field against its allowed set (or parse with a schema), and reject early — ideally before consuming any rate-limit quota.
Backed by OWASP LLM10:2025 Unbounded Consumption / Denial of Wallet (genai.owasp.org): Unrestricted user input to a model endpoint enables resource-consumption abuse: attackers drive paid inference with inputs the application never intended to serve.
Warning
A deterministic image call is served from a request handler with no result cache — identical inputs are recomputed and re-billed.
app/generate/route.ts:40rule result-cache-candidateconfidence medium
Estimate the full price of the call is paid again for every duplicate request
The same input returns the same output, so every repeat (a retry, a refresh, two users asking for the same thing) is a second full charge for a result you already paid for.
Fix: Key a cache on a hash of the inputs that determine the output and read it before calling the provider; store the result with a TTL. A cache store already present in the repo is usually enough.
Backed by GPTCache: An Open-Source Semantic Cache for LLM Applications (Bang, 2023) (aclanthology.org): Caching model responses keyed on input similarity removes redundant paid calls; repeated or near-identical requests are a large fraction of production traffic.

Spend map where the calls live, ranked by estimated cost per 1k calls

File	Sites	Models	Runs in	Findings	Est. per 1k calls
app/generate/route.ts	1	unresolved	request handler	5	not priced

Per-1k figures use the same token assumption as the exposure above; a file whose model is not named in code is not priced.

Models usage and right-sizing candidates

Model	Sites	Est. per call	Est. per 1k calls, all sites
(not statically resolvable)	1	not priced	not priced

No model was flagged as larger than its task needs.

Research behind this report

Replicate / OpenAI / Azure async job APIs (primary provider docs) (replicate.com): Async generation jobs bill for the full run once submitted, independent of whether the result is ever fetched; providers expose an explicit cancel endpoint precisely so an abandoned job can be stopped.
Rules: orphaned-async-job
When Agents Do Not Stop: Uncovering Infinite Agentic Loops in LLM Agents (arXiv:2607.01641): Cost exhaustion is a core agentic-loop harm; a spend guard is the backstop.
Rules: no-budget-guard
OWASP LLM10:2025 Unbounded Consumption / Denial of Wallet (genai.owasp.org): A per-user quota that is absent in the default configuration provides no protection: unbounded consumption is governed by the deployed configuration, not by the presence of guard code.
Rules: guard-conditional-on-env, unvalidated-input-to-paid-call
GPTCache: An Open-Source Semantic Cache for LLM Applications (Bang, 2023) (aclanthology.org): Caching model responses keyed on input similarity removes redundant paid calls; repeated or near-identical requests are a large fraction of production traffic.
Rules: result-cache-candidate

Ship Check of an agent's change

From npx intentiai check on a small seeded app. The findings come from deterministic checks only; no model was asked, so there is no agent summary on this sample.

What this change does

invoice-sync · HEAD (uncommitted) · 5 files, +25 / -5 · Sep 23, 2026, 7:27 PM UTC

10 things to deal with before this goes out.

Critical 7High 3Medium 3

Risky areas touched1 migrationusers tablesync.tsenv STRIPE_WEBHOOK_SECRET

Also touched1 test file

Top risks

1Critical
Column legacy_status is dropped from users while the running app may still read it
prisma/migrations/001_add/migration.sql:1check migration-safetyconfidence high
Removing legacy_status is irreversible and takes an exclusive lock on users. Any code still reading legacy_status — including the version of your app that is running right now, and anything outside this repo — starts erroring the moment the migration lands.
Do this: Split this into two deploys. First ship the code that no longer uses legacy_status; once that is live everywhere, ship a second migration that drops it. GitLab uses a three-release process for exactly this.
When this is not a problem: Safe when the deploy is all-at-once with downtime, or when nothing outside this repo reads the table.
Evidence (1 line)
- prisma/migrations/001_add/migration.sql:1
```
ALTER TABLE users DROP COLUMN legacy_status
```
2Critical
Index built without CONCURRENTLY on orders
prisma/migrations/001_add/migration.sql:2check migration-safetyconfidence high
Building an index normally blocks every write to orders until it finishes. On a large table that is minutes, and every insert or update in that window — signups, orders, messages — waits or times out.
Do this: Use CREATE INDEX CONCURRENTLY, and run it outside a transaction (most ORMs wrap migrations in one, so this usually needs its own migration file).
When this is not a problem: Harmless on a new or tiny table, where the build is instant.
Evidence (1 line)
- prisma/migrations/001_add/migration.sql:2
```
CREATE INDEX orders_customer_idx ON orders (customer_id)
```
3Critical
New column on users has a computed default — rewrites the whole table
prisma/migrations/001_add/migration.sql:3check migration-safetyconfidence high
A constant default is instant on modern Postgres, but a computed one like now() or gen_random_uuid() has to be evaluated for every existing row. That rewrites the table under a lock that blocks all reads and writes to users.
Do this: Add the column with no default, backfill in batches, then set the default for new rows.
Evidence (1 line)
- prisma/migrations/001_add/migration.sql:3
```
ALTER TABLE users ADD COLUMN signup_day date DEFAULT now()::date
```

Check these before you deploy

Split this into two deploys. First ship the code that no longer uses legacy_status; once that is live everywhere, ship a second migration that drops it. GitLab uses a three-release process for exactly this.
Use CREATE INDEX CONCURRENTLY, and run it outside a transaction (most ORMs wrap migrations in one, so this usually needs its own migration file).
Add the column with no default, backfill in batches, then set the default for new rows.
Require an authenticated session at the top of the handler in src/api/admin.ts, check that the caller owns the record it touches, and add a rate limit. If it is genuinely meant to be public, add a comment saying so.
Add STRIPE_WEBHOOK_SECRET to .env.example and to whatever actually sets your production environment, and fail fast at startup if any required variable is missing.
Verify the provider's signature header against the raw request body before reading the payload in src/api/webhook.ts (for Stripe, stripe.webhooks.constructEvent with the endpoint secret), and reject the request when it fails.

Taken from the fixes of the findings above; no agent review was run.

Files, ordered by risk

Worst risk	File	Kind	Lines	Findings
Critical	src/api/webhook.ts	source	+13 / -0	5
Critical	prisma/migrations/001_add/migration.sql	migration	+3 / -0	4
Critical	src/pay.test.ts	test	+1 / -4	2
Critical	src/api/admin.ts	source	+7 / -0	1
High	src/jobs/sync.ts	source	+1 / -1	1

Everything else

Show 10 more findings, with evidence

4Critical
New endpoint in admin.ts has no auth check
src/api/admin.ts:4check auth-surfaceconfidence medium
Nothing in this file identifies or limits the caller, so anyone who finds the URL can call it as often as they like. If it reads data, that data is public; if it writes or costs money, they are spending yours. Missing authorization on new surfaces is the single fastest-growing defect class in AI-written code.
Do this: Require an authenticated session at the top of the handler in src/api/admin.ts, check that the caller owns the record it touches, and add a rate limit. If it is genuinely meant to be public, add a comment saying so.
When this is not a problem: Auth is often applied globally in middleware, a route group, or a gateway this check cannot see. Verify before treating it as exposed.
Evidence (1 line)
- src/api/admin.ts:4
```
export const admin = Router().delete("/api/admin/users/:id", async (req, res) => {
```
5Critical
STRIPE_WEBHOOK_SECRET is read by the code but set nowhere
src/api/webhook.ts:4check env-driftconfidence medium
The code now needs STRIPE_WEBHOOK_SECRET. Nothing in this repo declares it, so the deploy will start and then fail at the moment that code first runs — often on a specific user action rather than at boot, which makes it look like a random error.
Do this: Add STRIPE_WEBHOOK_SECRET to .env.example and to whatever actually sets your production environment, and fail fast at startup if any required variable is missing.
When this is not a problem: Your platform may inject these outside the repo (a dashboard, a secret manager). This check only sees files.
Evidence (1 line)
- src/api/webhook.ts:4
```
STRIPE_WEBHOOK_SECRET
```
6Critical
Webhook in webhook.ts trusts payloads without checking the signature
src/api/webhook.ts:6check auth-surfaceconfidence medium
Webhooks have to be public, so the only thing proving a request came from the provider is its signature. Without that check, anyone who finds the URL can post a fake event — a paid invoice, a completed checkout, a cancelled subscription — and this code will act on it.
Do this: Verify the provider's signature header against the raw request body before reading the payload in src/api/webhook.ts (for Stripe, stripe.webhooks.constructEvent with the endpoint secret), and reject the request when it fails.
When this is not a problem: Verification may happen in shared middleware this check cannot see.
Evidence (1 line)
- src/api/webhook.ts:6
```
export const webhook = Router().post("/api/webhooks/stripe", async (req, res) => {
```
7Critical
3 assertions removed from pay.test.ts
src/pay.test.ts:7check test-integrityconfidence high
The tests still run but check 3 fewer things. A green run no longer proves what it did before this change.
Do this: Compare src/pay.test.ts against its previous version and restore any assertion that was removed to make a failure go away, rather than because the behaviour changed.
When this is not a problem: Correct when a test was genuinely simplified, parameterised, or moved into a helper in the same change.
Evidence (3 lines)
- src/pay.test.ts:7 removed; line number is from before this change
```
expect(charge(500).amountCents).toBe(500);
```
- src/pay.test.ts:10 removed; line number is from before this change
```
expect(() => charge(0)).toThrow();
```
- src/pay.test.ts:11 removed; line number is from before this change
```
expect(() => charge(-1)).toThrow();
```
8High
stripe-webhook-retry-helper is imported but not in any manifest
src/api/webhook.ts:2check dependency-riskconfidence medium
The code imports stripe-webhook-retry-helper, and nothing installs it. Either the install step was skipped — in which case this crashes on the first run in a clean environment — or the package does not exist and the agent invented the name. Invented names repeat across runs often enough that attackers register them, so installing one blindly is its own risk.
Do this: Check each name resolves to a real, maintained package before installing it, then add stripe-webhook-retry-helper to the manifest and commit the lockfile.
When this is not a problem: Monorepo path aliases, workspace packages and generated modules can look undeclared without being wrong.
Evidence (1 line)
- src/api/webhook.ts:2
```
stripe-webhook-retry-helper
```
9High
Scheduled job runs 288x more often
src/jobs/sync.ts:2check ops-deltasconfidence high
This job went from about 1 to about 288 runs a day. Everything it does — database queries, API calls, any paid service it touches — multiplies by the same factor, continuously, whether or not anyone is using the app.
Do this: Confirm the new schedule "*/5 * * * *" is intended. If the job is only meant to catch up on missed work, keep the old interval and make the job process a backlog instead.
Evidence (1 line)
- src/jobs/sync.ts:2
```
cron.schedule("*/5 * * * *", () => syncInvoices());
```
10High
Test run narrowed with .only in pay.test.ts
src/pay.test.ts:4check test-integrityconfidence high
A .only marker makes the runner execute just this test and silently ignore every other test in the file. CI will be green because almost nothing ran.
Do this: Remove the .only marker before committing so the whole suite runs again.
When this is not a problem: A skip may be a deliberate, documented quarantine for a known-flaky test.
Evidence (1 line)
- src/pay.test.ts:4
```
describe.only("charge", () => {
```
11Medium
Migration sets no lock timeout
prisma/migrations/001_add/migration.sql:1check migration-safetyconfidence high
If this migration cannot get its lock immediately it waits, and every query that arrives behind it queues too. One slow migration becomes a site-wide stall rather than a failed deploy.
Do this: Set a short lock_timeout (a few seconds) at the top of the migration so it fails fast and retries instead of blocking traffic.
When this is not a problem: Your migration runner may set this globally; check its configuration before acting.
12Medium
Error caught and discarded in webhook.ts
src/api/webhook.ts:9check swallowed-errorsconfidence medium
When this fails, the failure disappears. The request returns as if it worked, the user sees a success state built on nothing, and there is no error to find later — which is why this class of bug is usually discovered from a customer report rather than from monitoring.
Do this: Either handle the error meaningfully or re-raise it at src/api/webhook.ts:9. If it truly is safe to ignore, say why in a comment so the next reader does not have to guess.
When this is not a problem: Genuinely optional work (best-effort telemetry, cache warming) is fine to swallow.
Evidence (1 line)
- src/api/webhook.ts:9
```
} catch (e) {
```
13Medium
1 unfinished marker left in the change
src/api/webhook.ts:11check unfinished-workconfidence high
Something here is a placeholder rather than a working implementation. If the task was reported as complete, this is the gap between what was claimed and what exists — and it will surface as a runtime failure or a silently wrong result rather than a build error.
Do this: Open each marker and either finish it or delete the dead path. Start with src/api/webhook.ts:11.
When this is not a problem: A TODO can be a deliberate, tracked note rather than an unfinished edit.
Evidence (1 line)
- src/api/webhook.ts:11
```
// TODO: verify the Stripe signature before trusting the payload
```

Cost audit of roomGPT

Estimated exposure

Findings ranked by severity

Spend map where the calls live, ranked by estimated cost per 1k calls

Models usage and right-sizing candidates

Research behind this report

Ship Check of an agent's change

Top risks

Column legacy_status is dropped from users while the running app may still read it

Index built without CONCURRENTLY on orders

New column on users has a computed default — rewrites the whole table

Check these before you deploy

Files, ordered by risk

Everything else

New endpoint in admin.ts has no auth check

STRIPE_WEBHOOK_SECRET is read by the code but set nowhere

Webhook in webhook.ts trusts payloads without checking the signature

3 assertions removed from pay.test.ts

stripe-webhook-retry-helper is imported but not in any manifest

Scheduled job runs 288x more often

Test run narrowed with .only in pay.test.ts

Migration sets no lock timeout

Error caught and discarded in webhook.ts

1 unfinished marker left in the change