IntentiAI

Choosing a cheaper LLM model: right-sizing, routing, cascades

How to move LLM calls to cheaper models without losing quality: evaluate on your own data, then right-size, route or cascade. What FrugalGPT and RouteLLM show.

5 min read

The way to choose a cheaper LLM model is to test it against the model you use now, on a sample of your own real inputs, and compare cost per successful result rather than price per token. For many short, well-defined tasks (classification, extraction, routing, short rewrites) a small model does as well as a frontier model at a fraction of the price. For tasks where it does not, a router or a cascade can send only the hard requests to the expensive model.

There are three approaches, in increasing order of complexity: right-size each call site to one cheaper model, route each request to a model based on predicted difficulty, or cascade by trying a cheap model first and escalating when its answer fails a check.

Start with an inventory

List every call site with its model, what it does, and roughly how often it runs. Most applications have a few high-volume call sites doing narrow jobs (tagging, intent detection, JSON extraction, titles) and a few low-volume ones doing open-ended reasoning. The high-volume narrow jobs are where right-sizing pays. A frontier model returning a 10-token label thousands of times an hour is the typical find.

Build a small evaluation set

  1. Sample 100 to 300 real inputs for the call site, including the awkward ones from logs and support tickets.
  2. Record the current model's outputs as a baseline, and have a person mark which are correct. The current model is not ground truth.
  3. Write a scorer: exact match for labels, field-level match for extraction, a rubric or a pairwise judge for free text.
  4. Run the candidate models with the same prompt and settings, then again with a prompt adjusted for the smaller model. Small models often need more explicit instructions or a few examples.
  5. Compare accuracy, cost per correct answer, and latency at the 95th percentile, not the average.
Cost per successful result (illustrative numbers):
  Model A: $0.0100 per call, 96% correct  ->  $0.0100 / 0.96 = $0.0104 per correct answer
  Model B: $0.0008 per call, 91% correct  ->  $0.0008 / 0.91 = $0.0009 per correct answer

B is about 12x cheaper per correct answer. Whether 5 points of accuracy
is acceptable is a product decision, not an arithmetic one.

Right-sizing: one cheaper model per call site

If the small model meets the bar on your eval set, switch the call site and keep the eval set as a regression test for the next model change. This is the simplest change and usually the largest saving per hour of work. Pin model versions, because an alias that silently moves to a new snapshot can change both quality and price.

Routing: pick a model per request

A router looks at each request and predicts whether a cheap model will handle it. RouteLLM (Ong et al., ICLR 2025, arXiv:2406.18665) trains routers on preference data and reports cost reductions of over 2x in some cases without losing response quality, including up to 85% lower cost on MT Bench while keeping 95% of GPT-4's performance. The paper also found that its routers transferred to model pairs they were not trained on, and the code is open source. MixLLM (NAACL 2025) reports reaching 97.25% of GPT-4 quality at 24.18% of the cost in its setting.

Treat those as "up to" results on specific benchmarks with the models and prices of the time. Your traffic has a different mix of easy and hard requests, and that mix determines the saving. A router is also another component to monitor: if its threshold drifts, cost or quality moves with it.

Cascades: try cheap first, escalate on failure

A cascade calls the cheap model, checks the answer, and calls the expensive model only when the check fails. FrugalGPT (Chen, Zaharia and Zou, TMLR 2024, arXiv:2305.05176) reports matching the best individual model with up to 98% cost reduction. That figure is best-case, on a set of narrow benchmarks, at 2023 prices. The paper's more durable contribution is the taxonomy: adapt the prompt, approximate the model (caching, smaller models), and cascade.

import json

def extract_invoice(text: str) -> dict:
    cheap = call_model(SMALL_MODEL, EXTRACT_PROMPT, text, max_tokens=400)
    parsed = try_parse(cheap)
    if parsed is not None and passes_checks(parsed, text):
        return parsed

    strong = call_model(LARGE_MODEL, EXTRACT_PROMPT, text, max_tokens=400)
    return json.loads(strong)

def passes_checks(inv: dict, source: str) -> bool:
    # Cheap, deterministic checks beat asking another model to grade the answer.
    if not inv.get("invoice_number") or inv["invoice_number"] not in source:
        return False
    line_sum = round(sum(item["amount"] for item in inv.get("items", [])), 2)
    return abs(line_sum - inv.get("total", -1)) < 0.01

The expected cost of a two-step cascade is the cheap call plus the escalation rate times the expensive call. With illustrative costs of $0.001 and $0.012 per call and a 20% escalation rate, that is 0.001 + 0.2 * 0.012 = $0.0034, about 72% less than always calling the large model. At a 70% escalation rate it is $0.0094, and the extra latency on escalated requests may no longer be worth it. Measure the escalation rate before building the cascade.

Caveats that apply to all three

  • The check is the hard part. Cascades work well when correctness can be verified cheaply (schema validation, arithmetic, a value present in the source). When it cannot, the verifier is either another model call, which eats the savings, or a guess.
  • Cheaper per token is not always cheaper per task. Some smaller reasoning models produce many more tokens to reach the same answer. Measure tokens per task, not just price.
  • Latency moves both ways. Small models are usually faster, but a cascade adds a full round trip on every escalation.
  • Prices and models change quickly. Re-run the eval set when you change models or when providers change prices. Savings computed last quarter may not hold.
  • Complex scaffolds are a model choice too. Kapoor et al. ("AI Agents That Matter", arXiv:2407.01502) found a complex agent design costing roughly 50 times more than a simple baseline with no significant accuracy gain on their benchmark. Check whether the architecture, not only the model, is oversized.

One more pattern to avoid: a fallback that switches to a more expensive model after an error and never switches back. A single timeout then changes the cost of every later call in the process. Use the fallback for the failed request only. See sticky-fallback-model.

Model choice is one lever among several. How to reduce LLM API costs covers the others, several of which (output caps, resent history, batching) save money without any quality trade-off at all.

How IntentiAI helps

The model-oversized rule flags call sites that use a frontier model with a short, capped output, which is the signature of a simple job, and suggests the cheaper sibling in the same family. The finding cites FrugalGPT and RouteLLM with the benchmark caveats above, and the report states the model and call volume it assumed for any dollar estimate. Run npx intentiai in the repository to get the full inventory of call sites and models.

It does not evaluate quality. A static scan cannot know whether the smaller model is good enough for your data; that answer comes only from an eval set like the one described here. Treat each finding as a candidate for testing, not a verdict. See the cost audit and the docs.

More guides

Choosing a cheaper LLM model: right-sizing, routing, cascades | IntentiAI