VAMSHI.

Agentic Engineering

Optimize the Workflow Before the Token Bill

Reduce inference cost in the order that protects product quality: remove calls, route work, shrink context, cache, batch, and then tune.

5 min read
inference optimization ai economics llm infrastructure model routing ai product strategy
A cost waterfall showing savings from removing calls, routing tasks, reducing context, caching, constraining output, and batching.

The most expensive token is often part of a model call the product never needed to make.

Teams trim prompts while the workflow still classifies the same ticket twice, retrieves an entire account history, reruns stable analysis, asks a large model to calculate exact totals, and launches three graders for a low-risk draft.

Start with the workflow. Token optimization comes later.

Measure cost per accepted outcome

Cost per million tokens is a provider price. It is not your unit economics.

The useful measure is:

total workflow cost
────────────────────────
accepted, verified outcomes

The numerator includes failed runs, retries, retrieval, model calls, tool infrastructure, parallel agents, graders, and human review. The denominator excludes drafts nobody accepted.

A cheap run that fails twice may cost more than one strong run that completes. A more expensive planning step may reduce total cost if it prevents a broad patch and an hour of review.

Optimize in the order of leverage

1. Remove calls

Look for work that should be code, a rule, or a lookup:

  • exact arithmetic
  • schema validation
  • known policy thresholds
  • duplicate detection
  • stable formatting
  • status checks

Do not ask a language model whether a required field is empty.

Also remove duplicated judgment. If one grounded step already classified the issue, downstream prompts should consume that structured result instead of asking again.

2. Route by task and stage

Reserve stronger models and deeper reasoning for ambiguity, planning, and high-consequence decisions. Use faster routes for bounded transformations. Let deterministic systems own exact truth.

The routing policy should be eval-gated; “smaller model” is not a synonym for “same product, cheaper.”

3. Reduce context

Large contexts make prototypes easy and bills opaque.

Retrieve only the evidence needed for the current step. Deduplicate instructions. Store confirmed facts in structured state. Summarize old task history while preserving decisions and unresolved constraints. For code, use repository search and dependency maps rather than attaching every file.

Context is a budget. Every item should have a reason to be present.

4. Cache stable work

Reuse results whose meaning has not changed:

  • normalized source documents
  • embeddings and retrieval indexes
  • repository maps at a known revision
  • account summaries with a freshness timestamp
  • repeated prompt prefixes supported by provider caching

Cache keys must include the inputs that affect correctness. A stale summary is cheap until it causes an expensive mistake.

OpenAI’s prompt-caching guide is useful here: organize stable shared context so the platform can reuse it, then measure actual cache behavior rather than assuming a long prompt is cached well.

5. Constrain the output

Ask for the artifact the next step needs.

A classifier should return the schema, not a paragraph explaining classification. A coding agent should return a patch and evidence packet, not a diary. Shorter output also reduces review time, which is often more expensive than inference.

6. Batch work that does not need an immediate answer

Offline enrichment, nightly evals, bulk classification, and report preparation may tolerate asynchronous processing. Provider batch or flexible processing modes can trade latency for lower cost or better throughput.

Do not batch an interactive step merely because the API is cheaper. The user experience owns the latency budget.

7. Put budgets on retries and fan-out

Parallel agents, search branches, fallback models, and graders multiply calls quickly.

Set limits per task class:

max tool retries: 2
max routing escalations: 1
max parallel branches: 3
max grader calls: 1 for low-risk, 2 for high-risk
stop if no milestone changes within budget

Budgets force the system to surface ambiguity instead of spending through it.

A support workflow before and after

An initial billing-support agent may do this:

large model classifies ticket
→ entire account history enters context
→ large model identifies disputed invoice
→ large model calculates difference
→ large model drafts response
→ two graders review every response

A better workflow looks like:

fast classifier
→ deterministic account lookup
→ retrieve disputed invoice and relevant events
→ code calculates difference
→ policy rules decide routine vs ambiguous
→ fast model drafts routine response
→ strong model or human reviews only exceptions

The main saving comes from changing the shape of the work. Prompt trimming is now a useful final pass, not the strategy.

Guard the product contract

For every optimization, watch task success, error severity, escalation rate, reviewer time, p50 and p95 latency, and total cost per accepted outcome.

Plot the tradeoff. Some workflows promise an immediate answer. Others promise careful reasoning. A cost change that breaks the promise is not an optimization.

The goal is not minimum inference spend. It is minimum total cost for the quality the product has promised.

Further reading