A cheaper model response can create a more expensive support case. If a customer needs another explanation, an engineer corrects the answer, or the agent retries several tools, the token saving may disappear before the task ends.
That is the test I would apply to Model Router in Microsoft Foundry. The relevant comparison is the total cost of an accepted business outcome at an agreed quality level. Selecting a model dynamically is an implementation choice whose value must be measured.
Establish the fixed-model baseline first
Microsoft’s May 19 router evaluation guide describes an open-source approach that measures quality, cost, latency, and selected-model distribution. It explicitly includes router input-prompt billing alongside underlying inference costs. That is a useful starting point for a defensible comparison.
Before running it, define the baseline deployment and workload. A baseline should reflect what the application would otherwise use, with the same retrieval sources, tools, and output requirements. Comparing a carefully configured router with an artificially weak baseline produces a promotional result rather than an engineering decision.
Microsoft’s June model lifecycle guide places selection inside a broader operating loop. Use that approach to separate model capability from application correctness. A good answer still needs valid citations, appropriate permissions, and compatible tool arguments.
What earlier routing research demonstrates
The FrugalGPT study, first released in 2023, evaluated learned cascades across tasks including news classification, reading comprehension, and scientific question answering. It reported matching the best individual model with up to 98% lower cost, or improving accuracy over GPT-4 by 4% at the same cost in its experiments.
Those are separate research comparisons under historical models, prices, and datasets. They are not Foundry Model Router results, and the reported accuracy percentage should not be relabelled as percentage points without checking the underlying comparison.
The research supports testing workload-dependent selection. It does not justify putting a 98% reduction into an enterprise budget forecast. Build the forecast from measured task mix, current pricing, and actual downstream work.
Build a dataset that resembles the queue
Imagine a hypothetical employee-support assistant handling routine policy questions, complex eligibility questions, and requests requiring a business-system lookup. The easy questions may favour a lower-cost path. The complex cases may require stronger reasoning or escalation.
Sample the actual queue by task class and include difficult cases deliberately. Preserve a separate holdout set for the final comparison. Do not repeatedly tune routing rules against the same small set and then call its result independent validation.
Include long prompts, missing evidence, contradictory sources, multilingual requests where relevant, and tool failures. The Lost in the Middle research showed why context use deserves testing rather than being inferred from maximum input size. A model accepting the prompt does not establish that it used the important passage.
Compare outcomes before averaging costs
For each request, retain the task identifier, configuration, selected model where available, accepted result, latency, and cost components. Distinguish an answer that was generated from one that met the review criteria.
Report quality by task class before reporting an overall average. A small improvement on frequent easy requests can hide a serious regression on infrequent eligibility decisions. The business owner should decide which classes have hard acceptance requirements.
Calculate cost per accepted task by dividing total processing and correction cost by accepted outcomes. Keep rejected responses in the numerator. Otherwise the most error-prone configuration may appear cheapest because its failures were removed from the calculation.
Account for routing and recovery
Include input and output inference, routing charges, retrieval, tool calls, retries, monitoring, evaluation, and human correction. Show which components were measured and which were estimated.
Track the distribution of selected models. A change in that distribution may explain a cost or latency movement even when the application code stayed the same. Preserve enough configuration detail to compare runs after the available model pool changes.
Test what happens when an otherwise eligible model is unavailable or lacks the required capability. The fallback path must preserve the task contract. A substitute that cannot produce the expected structured output is not a successful fallback.
Keep the evaluator from deciding the result by accident
The May guide describes dual-ordered pairwise scoring to reduce position bias in model-based judgement. Use that as a useful check, then calibrate judgement against domain reviewers on representative cases.
Blind reviewers to the candidate where practical. Give them explicit criteria for correctness, evidence, completeness, and policy compliance. Keep evaluator version and prompts with the result, because changing the judge can change the apparent winner.
For transactional tasks, inspect the final system state as well as the response. A judge may prefer a polished explanation that accompanies an incorrect tool action. Deterministic checks should assess facts that can be verified directly.
Make a decision by workload class
A reasonable result may be mixed: the router suits routine questions, while a fixed deployment remains preferable for a specific constrained workflow. There is no requirement that one serving policy handle the entire application.
Set release thresholds before the experiment. Include permitted quality regression, latency limits, cost objective, and any task classes where regression is unacceptable. Label these as business choices rather than universal platform recommendations.
Stage the rollout with a fallback to the known-good path. Monitor accepted task quality and cost distribution after release, because production traffic may differ from the offline dataset.
Keep the economics honest over time
Re-evaluate when pricing, model eligibility, prompts, retrieval, or the task mix changes. A successful test describes a configuration and workload at a point in time.
The operating report should show where savings came from and where review or retry work increased. That allows the sponsor to decide whether the trade-off remains worthwhile.
Model Router can reduce manual model-selection work and may improve application economics. The business case becomes credible when the team can demonstrate lower total cost for accepted tasks while preserving the quality and behaviour those tasks require.
References
Microsoft: Router evaluation guide, May 19, 2026
Microsoft: Model lifecycle guide, June 2, 2026
