VAMSHI.

Agentic Engineering

Model Routing Is a Product Policy

Choose models by task stage, risk, quality target, latency budget, and cost ceiling—and change routes only through evaluation.

4 min read
model routing ai product strategy llm infrastructure agent architecture ai economics
A routing policy evaluates task stage, risk, evidence, latency, and cost before selecting a fast model, reasoning model, deterministic code, or human review.

“Which model should we use?” is usually the wrong unit of decision.

A claims workflow may need to classify an intake, extract fields, reconcile evidence, calculate a policy limit, draft an explanation, and escalate an ambiguous case. Those stages do not have the same quality requirement, latency budget, or consequence of error.

Model routing turns the product’s promises into an execution policy.

Route the stage, not the user

Consider a support agent handling a billing dispute.

StageBest defaultWhy
Detect language and ticket typeFast model or classifierHigh volume, bounded output
Retrieve account evidenceDeterministic toolsSource data must remain authoritative
Calculate invoice differenceCodeExact arithmetic is not a language task
Interpret conflicting recordsStrong reasoning modelAmbiguity carries real consequence
Check policy thresholdsRules and schemasAuditable, stable logic
Draft customer explanationFast capable modelNatural language with grounded facts
Approve credit above limitHumanAccountable business decision

One expensive model can perform most of these steps. That does not make it the best architecture.

Define the route from product constraints

A practical router considers:

  • task class and complexity
  • consequence of an error
  • required tool use or modality
  • amount and freshness of context
  • latency target
  • quality threshold
  • per-task cost ceiling
  • data and deployment constraints
  • evidence quality
  • current provider health

Keep the first version explicit. A routing table is easier to debug than a learned policy whose decisions nobody can explain.

For example:

IF task = classification AND schema_valid
  USE fast_route

IF evidence_conflict = true OR verifier_failed = true
  USE reasoning_route

IF action_risk = high
  REQUIRE human_review

The goal is not to predict which model will sound smartest. It is to meet a defined task contract.

Escalate on evidence, not vibes

Self-reported confidence is a weak escalation signal. Combine it with observable conditions:

  • required evidence is missing
  • retrieved sources disagree
  • a schema or invariant fails
  • the input falls outside the evaluated distribution
  • the first route repeats a tool call without progress
  • a verifier rejects the result
  • policy classifies the action as high impact

Escalation also needs a limit. Asking three progressively larger models the same underspecified question can turn ambiguity into cost without producing knowledge.

When the evidence is inadequate, the correct route may be to request data or stop.

Evaluate the routing system end to end

A benchmark score for each candidate model does not tell you whether the policy works.

Measure:

  • accepted task success by task class
  • p50 and p95 end-to-end latency
  • blended cost per accepted outcome
  • escalation and fallback rates
  • review time
  • failure severity
  • route stability over time

Compare the policy with simple baselines:

  1. Always use the strongest route.
  2. Always use the fastest route.
  3. Use one explicit two-tier escalation rule.

If a sophisticated router cannot beat those baselines on your eval set, it has added a control plane without adding value.

Research systems such as RouteLLM, RouterBench, and FrugalGPT show that routing and cascades can improve the quality-cost tradeoff. They also reinforce a practical point: the result depends on the task distribution and evaluation method. There is no universal route table.

Fallback is a product decision

When the primary route is unavailable, “try the next model” may be wrong.

For low-risk summarization, a smaller model may be an acceptable fallback. For a security-sensitive patch or high-value credit decision, the correct fallback may be cached evidence, a narrower read-only response, a human queue, or a safe stop.

Write the degradation behavior per stage:

classification unavailable → queue briefly, then use backup classifier
explanation model unavailable → return evidence without generated prose
high-risk reasoning unavailable → stop and request review

This is the operational form of a model exit plan: the product knows which capability it needs and which compromise it can expose.

Treat route changes like releases

Models, prices, latency, and availability change. Keep the policy configurable, versioned, observable, and easy to roll back.

Before changing a route, replay representative tasks. Inspect regressions by task class, not only the average. Estimate the new blended economics. Deploy gradually. Watch accepted outcomes and reviewer burden in production.

The best model is a moving target. The quality promise should not be.

Further reading