A dependency upgrade passes compilation and the agent answers a test question. The release reaches production, where a browser client fails to load, a background response resumes incorrectly, or telemetry costs change. The package installed successfully; the application contract still changed.

Foundry upgrades deserve the same release discipline as database clients and application frameworks, with an additional check for changes in model behaviour. Evaluate compatibility and task quality together, while keeping their causes distinguishable.

Upgrade a recorded configuration

Microsoft’s September roundup, published October 15 documents language-specific SDK changes rather than one universal upgrade path. For example, it identifies JavaScript browser-bundling fixes in version 2.7.1 and a .NET Projects line that remained beta. A stable package in one language does not establish equivalent feature maturity in another.

The same roundup calls out runtime defaults, including 100% default Azure Monitor trace sampling in AgentServer Core’s discussed release. That is a configuration detail, not a performance statistic. Review it against expected traffic, privacy requirements, and telemetry budget before rollout.

Create a baseline record containing package versions, runtime image, protocol, model deployment, prompts, tools, retrieval configuration, and policies. Include indirect dependencies and the deployment configuration used by the running service. A lockfile alone may not capture all of them.

Change one major dimension at a time where practical. If the immediate goal is to fix a client issue, hold the model configuration constant. If the goal is to compare a model, retain the known-good SDK path. This makes failures easier to attribute.

Test the interfaces the application actually uses

Build a compatibility matrix around real application paths: authentication, resource creation, streaming, tool invocation, structured output, file handling, and cancellation. Do not require every possible feature to pass if the application does not use it.

Run the client in its actual environment. A server-side test does not establish that browser bundling works. A local interactive credential does not establish that the deployed workload identity has the intended permission.

Inspect stream handling as a protocol, not just a text result. Confirm event order, partial output, tool events, final status, and error handling. A user interface that shows generated text but misses a terminal failure can misrepresent an unfinished operation as completed.

The September Agent Framework article explains that recovery can replay interrupted steps and that external effects still require idempotency. Verify the application’s write behaviour after interruption. Recovery support should not become a reason to remove reconciliation logic.

For a hypothetical procurement assistant, test a timeout after submitting a purchase request. The recovered agent should query the existing operation and return its confirmed state. Creating a duplicate request is a compatibility failure even if the new SDK handles reconnection exactly as documented.

Evaluate the business behaviour separately

Microsoft’s model lifecycle guide recommends comparing candidate configurations against workload-specific baselines. Use the same source snapshot, task inputs, and evaluation criteria for the candidate and current release.

Include representative ordinary work, known incidents, ambiguous requests, and cases that require refusal or escalation. Review correctness, evidence, policy compliance, tool arguments, and final business state. Keep important task classes separate in the report.

The Model Router evaluation guide provides a useful quality, cost, and latency comparison approach. If the application uses routing, record the selected-model distribution where available. Otherwise a change in routing behaviour can look like an SDK effect.

For model-based judging, retain the evaluator configuration and calibrate it against domain review. A change in evaluator behaviour should not silently redefine the release gate. Deterministic checks should handle schema validity, authorization boundaries, and verifiable transaction state.

Do not interpret a small clean test as proof that rare failures disappeared. As an illustrative calculation, if 300 independent, representative trials show zero occurrences of a particular failure, the one-sided 95% binomial upper bound is approximately 0.994%: one minus 0.05 raised to the power of one divided by 300. This is a statistical illustration, not a measured Foundry result.

The assumptions matter. Closely related prompts are not necessarily independent, and a convenient test set may not represent production. Use the calculation to explain uncertainty, not to assign a misleading assurance percentage to the whole application.

Stage the release around observable outcomes

Start in a non-production environment with synthetic or permitted test data. Verify migrations and configuration changes before sending production requests.

Use shadow evaluation only where its access and data handling are authorized. Keep side-effecting tools disabled or replaced with controlled test equivalents. A shadow run must not submit a second purchase request or send duplicate communication.

For a limited live rollout, monitor accepted tasks, review corrections, latency, tool failures, repeated actions, and total cost. Compare with the baseline over a meaningful task mix rather than using only request volume.

Set stop conditions before deployment. A wrong-authority action, repeated external write, or critical task regression may justify immediate containment even if average quality improved. The business owner should define those priorities with the application team.

Make rollback include unfinished work

Keep the prior application configuration available while evaluating the candidate. Confirm that required model deployments and supporting resources remain usable under the intended rollback plan.

Identify how active sessions, persisted state, and unfinished background work behave across versions. A prior runtime may not understand state written by the candidate. Resolve that compatibility question before relying on rollback during an incident.

Separate restoring a version from repairing business effects. Reverting code does not cancel an order or retract a message. Use the operation log to reconcile tasks that crossed the release boundary.

The release record should explain the reason for upgrading, evidence of compatibility, quality and cost results, remaining limitations, and the recovery plan. Assign an owner to the decision and a date to revisit temporary exceptions.

A useful upgrade delivers a specific improvement without losing control of the application’s behaviour. Newer versions create opportunities. Reproducible tests, staged exposure, and business-state verification determine whether the change belongs in production.

References

Microsoft: September Foundry roundup, October 15, 2026

Microsoft: Agent Framework interactive and resilient execution updates, September 24, 2026

Microsoft: Model lifecycle guide, June 2, 2026

Microsoft: Model Router evaluation guide, May 19, 2026

An invoice-processing agent needs access to the supplier registry and finance API. Then a document includes an upload link, and the agent attempts to follow it. Reviewing the agent’s approved tool names does not answer whether its runtime can send data to that destination.

Outbound connectivity is a separate architectural boundary. It deserves a policy and a test of its own, alongside tool permissions, identity, and validation of the information being sent.

Evaluate the preview within its documented scope

Microsoft’s September 24 egress walkthrough introduces a preview for hosted-agent network controls. The article explicitly states that it has no preview SLA and is intended for development and evaluation rather than production use. Keep that limitation visible when assessing it.

The walkthrough describes separate Audit and Enforced modes for destination policy. It also warns that Transform and Rewrite actions still operate in Audit. Use non-sensitive test values and review action behaviour before treating an Audit trial as harmless observation.

Scope conclusions to the documented hosted-agent HTTP and HTTPS path. A successful experiment should not be generalized to other protocols, unrelated agent types, or every possible route out of an enterprise environment.

The July hosted-agent GA announcement describes the runtime milestone. That does not promote this later egress preview to GA. Runtime availability and network-control maturity are distinct facts in the architecture decision.

Explain what an outbound rule can establish

Consider a hypothetical invoice agent that checks approved finance records. A destination policy can constrain where its HTTP requests go. It does not establish that an allowed request contains the correct invoice or that the recipient should receive every field.

Separate destination permission, caller authorization, and payload validity. The identity used for the finance API needs appropriate access. The application also needs to validate the customer, invoice, and requested operation.

This separation makes investigation clearer. If a permitted endpoint receives another customer’s document, the problem is not solved by adding more hostnames to a deny list. The application failed to bind the payload to the authorized business task.

Treat redirection and helper behaviour as part of the test plan. An apparently simple request can cause a library or remote endpoint to initiate another request. Identify what the selected runtime and policy actually observe and control rather than assuming the first URL describes the whole operation.

Build a controlled experiment with endpoint evidence

Use one approved test endpoint and one deliberately unapproved endpoint under the team’s control. Neither should receive production data. Record requests at the destination so the test has independent evidence of receipt.

Attach an explicit test policy to a defined agent version and preserve its identifier. Record the runtime version, policy version, and test input together. A policy document in configuration storage is evidence of intent, not proof of enforcement.

In Audit, inspect the policy decision evidence and compare it with what both endpoints received. In Enforced, repeat the same calls and verify the approved path still works while the unapproved destination receives nothing.

Do not infer a successful policy denial from an arbitrary error. DNS failure, an expired certificate, or an endpoint’s own access rejection can all prevent a request for unrelated reasons. Correlate runtime policy evidence with the destination log before calling the experiment successful.

Keep TLS verification enabled and follow the platform’s documented certificate handling. Disabling verification to make a test work changes the security conditions and can hide a separate configuration problem.

These are recommended test steps. They are not results from a test performed for this article. Publish measured outcomes only after the team has captured them in its own environment.

Pair network controls with agent security tests

The 2024 AgentDojo evaluation demonstrates why utility and security need separate measurement. In its GPT-4o defense comparison, a tool filter reduced targeted attack success from 57.69% without defense to 6.84%, while benign utility was 69.0% and 73.13% respectively. Those are historical benchmark results under the paper’s tested attacks, not egress-control results or estimates for Foundry.

A filtered tool set and a destination boundary address different parts of the problem. The research supports testing both legitimate work and attacker goals after adding a control. Blocking everything can look secure while making the application unusable.

Microsoft’s AI Red Teaming Agent announcement describes preview assessment for prohibited actions and sensitive-data leakage. Use application-specific scenarios alongside those capabilities: an invoice containing a malicious URL, a misleading tool response, or a request to send data to an unrelated account.

Include allowed endpoints that receive invalid payloads. An attacker may not need an unapproved destination if a legitimate integration can be misused. Business authorization and output validation remain necessary.

Turn the experiment into an operating decision

Assign an owner for policy changes, an owner for application validation, and an operator for failed connections. The runbook should distinguish an expected denial from a service outage and a certificate problem.

Before changing a policy, replay required business calls and deliberate negative tests. Check each active agent version that is expected to use the change. Do not assume that editing a policy instantaneously changes every running session.

Retain a record of why each destination is needed and what data the agent is permitted to send there. Periodically remove unused destinations through the normal review process. An allow list that grows without ownership eventually loses much of its value.

For the current preview, keep the experiment in a development environment and continue using the enterprise’s established production network controls. Evaluate the capability’s maturity again before making it a production dependency.

The desired evidence is concrete: required finance lookups complete, unapproved test destinations receive no request under enforcement, and allowed calls still obey identity and payload rules. That is a useful architecture result. A statement from the agent that it followed policy is not equivalent evidence.

References

Microsoft: Hosted-agent egress preview walkthrough, September 24, 2026

Microsoft: Hosted-agent GA announcement, July 9, 2026

Debenedetti and colleagues: AgentDojo, 2024, defense comparison Table 5

Microsoft: AI Red Teaming Agent, November 2025

A daily operations brief is straightforward until the agent misses a run, receives the same work twice, or reports on the wrong reporting window. The challenge is no longer generating a useful answer on demand. It is deciding what one unattended run is allowed to do and how to establish that the work is complete.

A schedule supplies a starting condition. The application needs a run contract that defines inputs, authority, time boundaries, outputs, recovery, and ownership. Establish that contract before enabling recurring execution.

Choose a task with a verifiable end

Microsoft introduced Routines in public preview in the June Build recap. The July 7 Foundry digest continued to describe Routines as preview for scheduled and triggered execution. Treat that as the maturity basis for an evaluation at this stage, rather than assuming a production service commitment.

Begin with a task whose outcome is easy to inspect. A report summarizing completed maintenance work is a better initial candidate than an agent that can independently change maintenance schedules and notify customers. The first produces evidence for review; the second creates several external commitments.

For a hypothetical maintenance-reporting agent, define the approved data sources, business unit, reporting period, output location, and recipients. Make those application configuration. The model can summarize the permitted records without choosing a broader scope or a new audience.

Give the run an explicit result type: completed, completed with declared gaps, awaiting review, failed, or superseded. A fluent report should not be the only success signal. The operator needs to distinguish “nothing happened” from “the agent could not retrieve the records.”

Define time as business data

Choose a business timezone and use explicit period boundaries. “Yesterday” is ambiguous when the service, database, and users operate in different zones. Persist the intended window with the run so a retry uses the same interval.

Decide whether records are selected by event time, completion time, or last modification. A maintenance job completed late but entered the following morning can otherwise disappear between reports. Consider a correction or reconciliation policy for late-arriving data.

Keep the reporting window independent from the actual start time. If the task begins late, it should still process the intended period unless the run policy explicitly says otherwise. Moving the window during recovery makes successive reports difficult to compare.

A catch-up decision also belongs in the contract. After an outage, should the application create each missed report, produce a consolidated report, or notify an operator? None is universally correct. Choose according to the consumer’s needs and record the choice.

Make repetition safe

Assign a stable business key to each intended result, such as business unit, report type, and reporting window. A retry should reconcile that key before producing another externally visible result.

The February durable-orchestration description provides background on workflows that wait and resume. Persistence helps coordinate execution, but the application still needs to establish whether an external write completed. Resuming a workflow is not proof that its effects occurred only once.

For the reporting example, write an internal versioned artifact before delivery. Mark which artifact is approved for distribution. If delivery times out, inspect its authoritative result before resending. Avoid letting a language-model explanation substitute for delivery evidence.

The 2024 τ-bench study assesses task outcomes through final database state and repeated trials. That evaluation principle transfers well to unattended runs: inspect the report and delivery state after retries, rather than judging only the agent’s final message.

Repeated execution can also produce a different summary from the same records. Keep the source snapshot or record versions needed to explain the difference. If a corrected report supersedes an earlier one, preserve that relationship rather than pretending both are independent daily results.

Bound authority and failure handling

Use a dedicated identity with only the access required by the recurring task. Revalidate permission at execution time. A schedule created by an authorized person should not become permanent permission to read data after access changes.

Restrict destinations through application configuration. A malicious instruction in a service record must not redirect the report or add a recipient. Treat retrieved text as information about the task, not authority to redefine it.

Set resource and retry bounds. Decide how long a run may continue, how many recoverable attempts are permitted, and when an operator receives an actionable failure. An agent should not keep searching indefinitely because a source system is unavailable.

Give incomplete data an explicit handling rule. If one approved source fails, the application may permit a report that clearly identifies the missing section, or it may require withholding the report. The business owner should make that decision before a live failure forces an improvised answer.

For any write-capable extension, separate proposal from execution and preserve the required approval. A timer can authorize starting analysis without authorizing the agent to make every change it recommends.

Prove useful unattended operation

Test duplicate invocation, a delayed start, a restart after artifact creation, a timeout after delivery, revoked access, and late-arriving records. Verify both the intended output and the absence of duplicate external effects.

Add a failure rehearsal for operators. They should be able to find the relevant run, identify the missing or uncertain step, and recover it without manually rebuilding the whole conversation. Keep source and output identifiers in the operational record.

Measure useful completed runs, incomplete reports, delivery errors, duplicate results, correction effort, and the time required to resolve failures. Track total cost per accepted report, including review and operation. A higher run count is not automatically a better outcome.

For the pilot, compare the generated brief with the existing manual report over equivalent periods. Ask the report’s consumers whether it changes a decision or saves meaningful effort. If they must reconstruct the source analysis every day, the automation has not yet earned its operating cost.

Keep the preview evaluation separate from the production commitment. Once the task boundary, recovery path, and value are established, the team can reassess the service’s release status and deployment requirements. A reliable recurring agent begins with a precise definition of one successful run.

References

Microsoft: Foundry Build Edition, June 2, 2026

Microsoft: June digest, published July 7, 2026

Microsoft: Durable agent orchestration, February 2026

Yao and colleagues: τ-bench, 2024

An invoice extractor returns the supplier, account reference, and total in valid JSON. The total is wrong. A downstream payment workflow accepts the structure and creates a payable record. The extraction looked technically successful until the business consequence appeared.

Document automation needs a decision about which values can proceed, which need review, and which should be rejected. Confidence and grounding help supply evidence for that decision. They do not replace business validation.

Keep the production and preview paths distinct

Microsoft’s August 12 Content Understanding update distinguishes a refreshed CU 1.0 GA API from CU 2.0 public preview. The preview introduces capabilities including synchronous Read and Layout APIs and more advanced document processing. Choose the API path according to the required feature and its maturity.

Do not infer production suitability from a model family’s availability alone. The analyzer, API version, processing mode, and model deployment form a combined configuration. Record and evaluate that combination.

The accompanying model-selection guide describes comparing deployments while holding the analyzer and inputs constant. That is valuable when the goal is to understand whether a model change actually improves a document workflow.

Read the reported efficiency results carefully

In its August guide, Microsoft reports up to 28% fewer total inference tokens and up to 25% lower full-inference LLM cost in tested configurations. It also reports confidence-score and grounding improvements measured using AUROC and grounding exact match. These are vendor-reported internal results, not a guaranteed reduction in the total cost of processing an invoice.

The separate August update associates the token result with GPT-4.1 and GPT-5.2 evaluations. The public summaries do not provide enough detail to assume the same improvement for every document type, schema, or deployment. Keep the “up to” qualification and test the actual configuration.

Token savings affect only part of the business case. Review effort, document preparation, storage, retries, exception handling, and correction of downstream records can dominate the total.

Begin with a narrow business document

Consider a hypothetical accounts-payable workflow for invoices from a defined supplier group. Start with required fields and explicit validity rules: supplier identity, invoice number, currency, line totals, tax, and payable amount.

Preserve the original document and a stable identifier for each processing attempt. Connect each extracted field to available source evidence. When a reviewer challenges a value, the interface should show the relevant document area rather than ask them to search the full file.

Use authoritative business systems to validate supplier and purchase-order details. A document can accurately contain a fraudulent account number. Extraction correctness and transaction legitimacy are separate checks.

Calibrate review by field and consequence

A single document-level score is rarely enough. The total amount may need stricter review than an optional description. A field’s business impact should influence the release rule alongside extraction confidence.

Build a labelled validation set from representative documents. Compare score ranges with observed correctness for each important field and document class. Do not treat a confidence value as a universal probability of correctness unless that interpretation has been established for the configuration.

Recheck calibration after model, analyzer, schema, or input changes. A threshold that was useful for clean supplier PDFs may behave differently on photographed documents or a revised invoice layout.

Use grounding to accelerate the review

Grounding should make the reviewer faster and more accurate by connecting a proposed value with its origin. Test whether the highlighted evidence supports the exact value and context.

An amount beside a subtotal label is not evidence for the final payable amount. A repeated supplier name in a footer may not identify the invoicing entity. Include these distinctions in the evaluation rubric.

The original DocVQA research established document-image question answering as a task requiring understanding of document content and layout. It is useful background for evaluating source location and interpretation, rather than assuming plain text extraction captures every important relationship.

Define the exception queue before increasing volume

Route incomplete documents, conflicting values, unsupported formats, and failed business checks into explicit states. Give reviewers the reason for routing and the fields that require attention.

Avoid sending every uncertain document through repeated model calls without a stopping rule. Repeated extraction can increase cost while producing several plausible answers to the same ambiguous input.

Distinguish a review correction from a new source document. If the supplier sends a revised invoice, retain version relationships and prevent the older record from re-entering processing as a separate payable.

Measure accepted documents and missed errors

Report field accuracy, critical errors that escaped review, review rate, reviewer time, and correctly completed documents. Show results by supplier group and input quality rather than relying on a single average.

Calculate total cost per accepted document. Keep rejected and reprocessed documents in the cost numerator. Report model inference savings separately so leaders can see whether they changed end-to-end economics.

Microsoft’s model lifecycle guidance recommends workload-based comparison and controlled upgrades. Apply that discipline to the full extraction configuration, not just the language model name.

Keep extraction separate from business execution

The extractor should propose structured information with evidence. A deterministic validation layer should apply arithmetic and business rules. A review layer should resolve exceptions. A separate authorized operation should create or update the downstream record.

Use idempotency and reconciliation at the integration boundary. A timeout after creating a payable must not lead to an automatic duplicate submission. The document identifier and business record should provide a way to establish the confirmed outcome.

For a first release, favour a smaller set of documents with well-defined exception handling over broad format coverage with unclear review rules. The practical goal is reliable automation with a manageable review burden. Grounding and confidence become valuable when they help the team make and verify those decisions.

References

Microsoft: Content Understanding updates, August 12, 2026

Microsoft: GPT-5 series and confidence guide, August 12, 2026

Mathew, Karatzas, and Jawahar: DocVQA, 2020

Microsoft: Model lifecycle guidance, June 2, 2026

A cheaper model response can create a more expensive support case. If a customer needs another explanation, an engineer corrects the answer, or the agent retries several tools, the token saving may disappear before the task ends.

That is the test I would apply to Model Router in Microsoft Foundry. The relevant comparison is the total cost of an accepted business outcome at an agreed quality level. Selecting a model dynamically is an implementation choice whose value must be measured.

Establish the fixed-model baseline first

Microsoft’s May 19 router evaluation guide describes an open-source approach that measures quality, cost, latency, and selected-model distribution. It explicitly includes router input-prompt billing alongside underlying inference costs. That is a useful starting point for a defensible comparison.

Before running it, define the baseline deployment and workload. A baseline should reflect what the application would otherwise use, with the same retrieval sources, tools, and output requirements. Comparing a carefully configured router with an artificially weak baseline produces a promotional result rather than an engineering decision.

Microsoft’s June model lifecycle guide places selection inside a broader operating loop. Use that approach to separate model capability from application correctness. A good answer still needs valid citations, appropriate permissions, and compatible tool arguments.

What earlier routing research demonstrates

The FrugalGPT study, first released in 2023, evaluated learned cascades across tasks including news classification, reading comprehension, and scientific question answering. It reported matching the best individual model with up to 98% lower cost, or improving accuracy over GPT-4 by 4% at the same cost in its experiments.

Those are separate research comparisons under historical models, prices, and datasets. They are not Foundry Model Router results, and the reported accuracy percentage should not be relabelled as percentage points without checking the underlying comparison.

The research supports testing workload-dependent selection. It does not justify putting a 98% reduction into an enterprise budget forecast. Build the forecast from measured task mix, current pricing, and actual downstream work.

Build a dataset that resembles the queue

Imagine a hypothetical employee-support assistant handling routine policy questions, complex eligibility questions, and requests requiring a business-system lookup. The easy questions may favour a lower-cost path. The complex cases may require stronger reasoning or escalation.

Sample the actual queue by task class and include difficult cases deliberately. Preserve a separate holdout set for the final comparison. Do not repeatedly tune routing rules against the same small set and then call its result independent validation.

Include long prompts, missing evidence, contradictory sources, multilingual requests where relevant, and tool failures. The Lost in the Middle research showed why context use deserves testing rather than being inferred from maximum input size. A model accepting the prompt does not establish that it used the important passage.

Compare outcomes before averaging costs

For each request, retain the task identifier, configuration, selected model where available, accepted result, latency, and cost components. Distinguish an answer that was generated from one that met the review criteria.

Report quality by task class before reporting an overall average. A small improvement on frequent easy requests can hide a serious regression on infrequent eligibility decisions. The business owner should decide which classes have hard acceptance requirements.

Calculate cost per accepted task by dividing total processing and correction cost by accepted outcomes. Keep rejected responses in the numerator. Otherwise the most error-prone configuration may appear cheapest because its failures were removed from the calculation.

Account for routing and recovery

Include input and output inference, routing charges, retrieval, tool calls, retries, monitoring, evaluation, and human correction. Show which components were measured and which were estimated.

Track the distribution of selected models. A change in that distribution may explain a cost or latency movement even when the application code stayed the same. Preserve enough configuration detail to compare runs after the available model pool changes.

Test what happens when an otherwise eligible model is unavailable or lacks the required capability. The fallback path must preserve the task contract. A substitute that cannot produce the expected structured output is not a successful fallback.

Keep the evaluator from deciding the result by accident

The May guide describes dual-ordered pairwise scoring to reduce position bias in model-based judgement. Use that as a useful check, then calibrate judgement against domain reviewers on representative cases.

Blind reviewers to the candidate where practical. Give them explicit criteria for correctness, evidence, completeness, and policy compliance. Keep evaluator version and prompts with the result, because changing the judge can change the apparent winner.

For transactional tasks, inspect the final system state as well as the response. A judge may prefer a polished explanation that accompanies an incorrect tool action. Deterministic checks should assess facts that can be verified directly.

Make a decision by workload class

A reasonable result may be mixed: the router suits routine questions, while a fixed deployment remains preferable for a specific constrained workflow. There is no requirement that one serving policy handle the entire application.

Set release thresholds before the experiment. Include permitted quality regression, latency limits, cost objective, and any task classes where regression is unacceptable. Label these as business choices rather than universal platform recommendations.

Stage the rollout with a fallback to the known-good path. Monitor accepted task quality and cost distribution after release, because production traffic may differ from the offline dataset.

Keep the economics honest over time

Re-evaluate when pricing, model eligibility, prompts, retrieval, or the task mix changes. A successful test describes a configuration and workload at a point in time.

The operating report should show where savings came from and where review or retry work increased. That allows the sponsor to decide whether the trade-off remains worthwhile.

Model Router can reduce manual model-selection work and may improve application economics. The business case becomes credible when the team can demonstrate lower total cost for accepted tasks while preserving the quality and behaviour those tasks require.

References

Microsoft: Router evaluation guide, May 19, 2026

Microsoft: Model lifecycle guide, June 2, 2026

Chen, Zaharia, and Zou: FrugalGPT, 2023

Liu and colleagues: Lost in the Middle, 2023

The monitoring dashboard is green. The agent’s requests completed, latency is within target, and no service returned an error. Yet a support specialist reports that the assistant recommended the wrong maintenance procedure.

This is an application incident even if every API returned success. Investigating it requires evidence about the retrieved source, selected tool, model configuration, and resulting recommendation. Infrastructure health is one part of that evidence.

Trace the business task across components

Microsoft’s March Agent Service announcement describes evaluation and monitoring capabilities as part of the platform. The June observability announcement adds a more detailed map of the development loop, including preview extensions for multi-turn evaluation, trace replay, and turning traces into datasets.

Do not assume that a foundational GA capability gives every new extension the same status. Identify which tracing, evaluation, and retention surfaces the application uses and verify them separately.

My recommendation is to start with an application task identifier. Propagate it through orchestration, retrieval, model requests, tool calls, and review outcomes. That makes it possible to ask what happened to one customer request across several systems.

A successful response is not a successful task

The 2024 τ-bench research found less than 50% task success for its tested function-calling agents and retail pass^8 below 25%. Those historical results are not a scorecard for current Foundry deployments. They demonstrate why repeated task-level checks can reveal weaknesses that a plausible conversation hides.

For a support agent, a success label should depend on the correct recommendation and permitted final state. If the agent cites an obsolete procedure, record that as a quality failure even when retrieval and generation completed normally.

Separate task success, policy compliance, and service availability. Combining them into one green indicator makes operational decisions harder. Different failure classes require different owners and remedies.

Reconstruct an illustrative incident

Imagine a hypothetical industrial support assistant that advises replacing a component. A reviewer later discovers that the equipment variant uses a different procedure. The trace shows the assistant retrieved a manual for the correct product family but the wrong variant.

That evidence points first to retrieval and metadata filtering. Changing the model’s general instructions might suppress the symptom without fixing the source selection. The investigation should establish where the wrong variant entered the workflow.

Retain the source document identifier, revision, retrieval filter, and tool arguments where permitted. Store the model and prompt configuration alongside the task. Without those details, a later replay may use changed sources and produce a misleadingly correct answer.

Collect evidence deliberately

Content capture can expose customer information, proprietary documents, and credentials accidentally returned by tools. Decide what the investigation genuinely needs before enabling broad logging.

Use identifiers, timing, configuration versions, and outcome labels as the default operational layer. Capture sensitive content only through a defined process with access controls and retention appropriate to its purpose.

Test the telemetry pipeline with representative sensitive inputs. Verify redaction where intended and check error paths, because exceptions often contain more information than normal logs. A configuration that protects successful requests may still leak content when a dependency fails.

Monitor quality by meaningful slices

Aggregate accuracy can improve while a critical equipment class becomes worse. Group results by task type, language, source family, model configuration, and relevant tool path. Keep small samples visible rather than reporting unstable percentages with false precision.

Use human review for cases where domain judgement is essential. Calibrate any model-based evaluator against that review. Track disagreement so an evaluator change does not silently redefine what the dashboard calls success.

Compare the current release with a stable baseline. Alert on a meaningful change in important task outcomes, then inspect whether the cause is a new model, revised source, changed tool, or traffic shift.

Design an incident response sequence

First contain the consequence. Pause a problematic write path or direct affected requests to review while preserving safe read-only assistance where appropriate.

Next identify the affected versions and tasks. Use the task identifier and configuration record to find potentially incorrect recommendations. A model deployment name alone may not describe every relevant component of the release.

Then verify the business impact. Determine whether the recommendation was merely generated, accepted by a reviewer, or acted upon. Those are different incident scopes and may require different communication and correction.

Finally, reconcile unfinished work. A rollback can restore a prior configuration, but it cannot determine whether a previous tool call completed. Use authoritative records to resolve uncertain outcomes.

Turn the incident into a release test

Build a regression fixture from the failure with permitted data. Include the confusing variants and the correct procedure. Run it against the proposed correction and the baseline.

Add neighbouring cases rather than a single memorized example. If a metadata filter fixes one product but excludes valid sources for another, the correction is incomplete.

The July hosted-agent GA announcement is useful context for managed operations, while the earlier Agent Framework release description explains orchestration foundations. Neither replaces an application-specific incident test. The release decision should use evidence from the actual support workflow.

Connect observability to cost and accountability

Calculate evaluation and telemetry cost alongside model and runtime spend. Sample ordinary traffic where appropriate, while deliberately retaining evidence for important failures under the chosen policy.

Report time to detect a quality regression, time to identify its cause, unresolved affected tasks, and recurrence after correction. Pair those measures with accepted business outcomes and human correction effort.

Assign an incident owner who can coordinate data, application, model, and platform teams. The purpose of a trace is to shorten the path from a bad outcome to a verified correction. A richer dashboard is useful only when it helps that process.

For the first rollout, choose one task family and walk through a controlled wrong-answer incident from detection to regression test. If the team can explain the failure and prove the correction, observability is doing practical work.

References

Microsoft: Agent Service evaluation and monitoring, March 16, 2026

Microsoft: Observability and ROI, June 3, 2026

Yao and colleagues: τ-bench, 2024

Microsoft: Hosted-agent GA announcement, July 9, 2026

Microsoft: Agent Framework architecture, February 19, 2026

An enterprise may know who owns its customer database and still struggle to name the owner of an agent that reads it, calls a pricing service, and prepares a customer offer. The agent crosses several teams’ boundaries in one task. Governance has to connect those responsibilities.

The first control I would establish is a clear answer to three questions: who owns the outcome, whose authority the agent uses, and who can stop its actions. A product inventory helps, but an entry in a catalogue is not an operating agreement.

Production hosting changes the operating obligation

Microsoft’s July 9 Foundry announcement identifies hosted agents as generally available alongside new models and deployment capabilities. Microsoft also reports that more than 100,000 organizations are building on Foundry. That is a vendor-reported adoption figure, not a measure of successful deployments or proof of governance maturity.

A managed hosting milestone can remove infrastructure work. It does not remove the application owner’s responsibility for incorrect quotes, unauthorized access, or unfinished transactions. Review the agent as a business application with delegated authority.

The April developer announcement and June hosted-agent update provide background on runtimes, identity, and controls. Read feature-specific maturity and deployment requirements rather than carrying a general GA label across every related capability.

Define the business action before the tool

Consider a hypothetical sales-support agent that assembles a renewal proposal. Its read operations retrieve contract terms and approved price information. Its write operations might save an internal draft or submit an offer for review. Sending the offer to the customer is a separate business action.

These operations have different consequences and should have separate authorization rules. A broad tool called “manage renewal” makes it difficult to distinguish harmless preparation from an external commitment.

Create narrow operations with explicit targets and validated arguments. The application should resolve the customer account from authenticated context and verified records. An instruction embedded in a document must not select a different tenant, customer, recipient, or authority.

Establish an owner for each control

The business owner defines permitted outcomes, escalation rules, and materiality. The application owner implements transaction validation and recovery. Identity teams establish access and credential lifecycle. Platform teams own deployment, monitoring, and configuration distribution.

Security teams should assess abuse paths, but they should not become the default owner of every unresolved application decision. A policy requiring approval is ineffective unless an application team implements it and a business team supplies authorized reviewers.

Record who can change prompts, tools, model deployments, and policy configuration. Treat those changes as application releases. A small configuration update can increase authority without changing the repository’s main code path.

Use adversarial research to improve test design

The 2024 AgentDojo study created 97 realistic tasks and 629 security test cases for agents operating on untrusted tool data. Those counts describe benchmark coverage, not the probability that a Foundry agent will be compromised. The study’s value here is its stateful testing approach.

A renewal agent should encounter a supplier document containing instructions to send information elsewhere, a tool response claiming that approval is unnecessary, and a retrieved note that conflicts with current policy. Score whether the agent completes the legitimate task and whether the application blocks prohibited actions.

Microsoft’s AI Red Teaming Agent announcement describes preview testing for risks including sensitive-data leakage and prohibited actions. Treat such tools as part of a wider evaluation programme. Confirm that the chosen tests reflect the actual tools and permissions of the application.

Keep policy enforcement outside model persuasion

Prompts should explain expected behaviour. Deterministic controls should enforce resource scope, recipient restrictions, financial bounds, and required approvals. A model-generated justification must not grant an exception.

Where a tool accepts a record identifier, validate that the caller may act on that record. Where it accepts an external destination, restrict that destination according to application policy. These checks remain necessary even when the agent usually chooses correct parameters.

Do not give every tool the same credential. Narrow identities make the effect of a compromised component easier to contain and investigate. Confirm the selected hosting path’s actual identity behaviour before assuming a deployment supplies the intended separation.

Plan for changes in authority

An agent that begins with read-only access can accumulate privileges through normal product development. Review new tools against the original task boundary. Adding an API may create a new disclosure route or business commitment.

Maintain an inventory of active agent versions, tool definitions, policy versions, and approved deployment destinations. Include dependencies outside Foundry where they participate in the workflow. The governance boundary should follow the business action across services.

Test removal of access as well as addition. A revoked role or disabled connection should lead to a clear failure or escalation. It should not cause the agent to search for a broader credential or alternate route.

Make stopping and recovery operational

Define how to stop new work, prevent pending writes, and reconcile actions already submitted. A deployment rollback does not reverse an offer that was sent or a customer record that was changed.

The runbook should identify the authoritative transaction log and the operator who can determine what happened. Preserve operation identifiers so incident responders can distinguish an attempted action from a completed one.

Exercise this process with a controlled failure. The test should show how unfinished work is located, how the customer impact is assessed, and how service resumes. Record the evidence rather than accepting a diagram as proof.

Give leaders a useful governance report

Report agents by business owner and permitted action scope. Include failed authorization tests, overdue remediation, unresolved transaction outcomes, and changes that increased autonomy. Separate these from general traffic and token usage.

For the renewal example, pair technical measures with incorrectly scoped proposals, review corrections, and confirmed customer impact. A dashboard with many successful model responses can still conceal a serious authority problem.

Begin expansion only when the current boundary is understood and tested. Foundry can provide useful hosting and governance building blocks. Enterprise control comes from connecting those capabilities to named owners, enforceable authority, and evidence about the resulting business actions.

References

Microsoft: July 9, 2026 Foundry announcement

Microsoft: Developer journey and identity foundations, April 22, 2026

Microsoft: Hosted-agent controls, June 2, 2026

Microsoft: AI Red Teaming Agent, November 2025

Debenedetti and colleagues: AgentDojo, 2024

A caller interrupts an appointment assistant halfway through a confirmation: “Actually, make that next Thursday.” The assistant has already started checking availability. The important question is whether it updates the intended date before booking, rather than how natural its voice sounds.

Voice applications combine conversational timing with business transactions. A technically impressive speech demonstration can still create the wrong booking, repeat a payment request, or leave a caller unsure whether anything happened. Design the transaction and conversation together.

Choose the correct Voice Live integration

Microsoft’s March Agent Service announcement described the Voice Live integration in preview. The June Build recap distinguishes GA Voice Live prompt agents from preview hosted voice agents. Those paths provide different deployment choices and should not share a blanket maturity claim.

The June hosted-agent announcement describes real-time connectivity through WebSocket and WebRTC, including custom runtime and framework options. Select a path according to the control the application needs over orchestration, channel behavior, and recovery.

Before committing to a stack, demonstrate the specific transport, model, identity, and tool configuration together. A supported speech feature in one path does not prove equivalent behavior in another.

What speech research does and does not establish

The 2022 Whisper research paper reports training on 680,000 hours of multilingual, multitask supervision. Its best zero-shot model achieved a 2.5% word error rate on LibriSpeech test-clean. Against a supervised model with similar in-distribution performance, it reported a 55.2% average relative error reduction across other evaluated speech datasets.

Those historical results concern Whisper under the paper’s conditions. They do not measure Voice Live latency or guarantee its transcription quality. Their relevance is the gap between a clean benchmark and performance across different recording conditions.

For a service workflow, the transcript metric also has a business limitation. Mishearing a filler word and mishearing an appointment date may contribute similarly to word error rate while having very different consequences. Test critical entities separately.

A practical appointment workflow

Imagine a hypothetical maintenance company taking calls to arrange equipment visits. The agent identifies the service request, checks eligible appointment windows, reads back the selected date and address, and creates the booking only after the required confirmation.

Represent the appointment as structured state outside the transcript. The date, timezone, site address, service type, and customer identity should have explicit validation status. Do not infer that an earlier confirmation still applies after one of those values changes.

Keep the availability lookup read-only. Put the booking operation behind a separate application check that verifies the final slot, current availability, and confirmation evidence. The speech model can lead the conversation while deterministic code enforces the booking contract.

Handle interruption at the right layer

When the caller interrupts, stop or adjust the spoken response promptly, then decide which application work remains valid. Stopping audio does not necessarily cancel a running API request.

If a lookup is still relevant, keep its result. If a write may have completed, reconcile it before submitting another one. When the caller changes the appointment date, invalidate the old proposal rather than merely replacing a phrase in the conversation history.

Use an operation identifier for each booking attempt. If the connection drops after submission, a resumed session should query the booking record and explain the verified outcome. It should not create a second reservation because the first confirmation was never heard.

Budget latency across the whole interaction

Measure the interval from the caller finishing an utterance to useful spoken feedback. Break it into capture and endpointing, model processing, tool work, and playback. Report both typical and slow interactions.

Separate an acknowledgement from a completed answer. Saying “I’m checking that” may improve conversational clarity, but it does not reduce the time until confirmed availability. Track those moments independently so the team does not optimize an artificial response-time metric.

Test realistic devices, background noise, variable connections, accents, and supported languages needed by the service. Use permitted recordings or synthetic fixtures with known ground truth. Keep the same test inputs when comparing configurations.

Protect critical details without making every turn tedious

Read back the details that materially affect the transaction. A date, service address, or charged amount may require explicit confirmation; a casual preference about wording usually does not.

Set a defined clarification path when the agent is uncertain. Repeating a question in different words can help, but repeated failure should trigger a useful alternative such as a text confirmation or transfer where the business supports it.

The handoff should include the verified task state and unresolved question. Avoid forcing the customer to repeat the entire interaction merely because the agent reached a boundary. Do not tell the caller a transfer or follow-up has occurred until the corresponding operation is confirmed.

Measure service outcomes

For the maintenance example, score correct appointments, wrong-date bookings, duplicate bookings, abandoned calls, and successful recovery after disconnection. Include the time staff spend correcting agent-created records.

Calculate cost per correctly completed request using speech, model inference, tool infrastructure, monitoring, and correction effort. Longer calls and repeated confirmations affect economics even when the per-token model rate is attractive.

Run the pilot against an existing service baseline on equivalent requests. Human review should check both the conversation and the final booking. A pleasant interaction with a wrong record is a failed task.

Launch with a bounded service promise

Start with one service type and a clear escalation path. Publish an operational runbook for unavailable scheduling systems, uncertain identity, caller corrections, and interrupted writes. Assign an owner for both conversation quality and transaction reliability.

Review failure clusters before expanding supported tasks. Some problems require a better model; others require a clearer question, stronger input validation, or a more reliable booking API.

A successful voice agent reduces the effort needed to complete a service request while preserving the caller’s control over the result. Natural speech helps people use it. Correct state, clear confirmation, and reliable recovery make it useful.

References

Microsoft: Agent Service and Voice Live, March 16, 2026

Microsoft: Foundry Build Edition, June 2, 2026

Microsoft: Hosted-agent updates, June 2, 2026

Radford and colleagues: Robust Speech Recognition, 2022