The monitoring dashboard is green. The agent’s requests completed, latency is within target, and no service returned an error. Yet a support specialist reports that the assistant recommended the wrong maintenance procedure.

This is an application incident even if every API returned success. Investigating it requires evidence about the retrieved source, selected tool, model configuration, and resulting recommendation. Infrastructure health is one part of that evidence.

Trace the business task across components

Microsoft’s March Agent Service announcement describes evaluation and monitoring capabilities as part of the platform. The June observability announcement adds a more detailed map of the development loop, including preview extensions for multi-turn evaluation, trace replay, and turning traces into datasets.

Do not assume that a foundational GA capability gives every new extension the same status. Identify which tracing, evaluation, and retention surfaces the application uses and verify them separately.

My recommendation is to start with an application task identifier. Propagate it through orchestration, retrieval, model requests, tool calls, and review outcomes. That makes it possible to ask what happened to one customer request across several systems.

A successful response is not a successful task

The 2024 τ-bench research found less than 50% task success for its tested function-calling agents and retail pass^8 below 25%. Those historical results are not a scorecard for current Foundry deployments. They demonstrate why repeated task-level checks can reveal weaknesses that a plausible conversation hides.

For a support agent, a success label should depend on the correct recommendation and permitted final state. If the agent cites an obsolete procedure, record that as a quality failure even when retrieval and generation completed normally.

Separate task success, policy compliance, and service availability. Combining them into one green indicator makes operational decisions harder. Different failure classes require different owners and remedies.

Reconstruct an illustrative incident

Imagine a hypothetical industrial support assistant that advises replacing a component. A reviewer later discovers that the equipment variant uses a different procedure. The trace shows the assistant retrieved a manual for the correct product family but the wrong variant.

That evidence points first to retrieval and metadata filtering. Changing the model’s general instructions might suppress the symptom without fixing the source selection. The investigation should establish where the wrong variant entered the workflow.

Retain the source document identifier, revision, retrieval filter, and tool arguments where permitted. Store the model and prompt configuration alongside the task. Without those details, a later replay may use changed sources and produce a misleadingly correct answer.

Collect evidence deliberately

Content capture can expose customer information, proprietary documents, and credentials accidentally returned by tools. Decide what the investigation genuinely needs before enabling broad logging.

Use identifiers, timing, configuration versions, and outcome labels as the default operational layer. Capture sensitive content only through a defined process with access controls and retention appropriate to its purpose.

Test the telemetry pipeline with representative sensitive inputs. Verify redaction where intended and check error paths, because exceptions often contain more information than normal logs. A configuration that protects successful requests may still leak content when a dependency fails.

Monitor quality by meaningful slices

Aggregate accuracy can improve while a critical equipment class becomes worse. Group results by task type, language, source family, model configuration, and relevant tool path. Keep small samples visible rather than reporting unstable percentages with false precision.

Use human review for cases where domain judgement is essential. Calibrate any model-based evaluator against that review. Track disagreement so an evaluator change does not silently redefine what the dashboard calls success.

Compare the current release with a stable baseline. Alert on a meaningful change in important task outcomes, then inspect whether the cause is a new model, revised source, changed tool, or traffic shift.

Design an incident response sequence

First contain the consequence. Pause a problematic write path or direct affected requests to review while preserving safe read-only assistance where appropriate.

Next identify the affected versions and tasks. Use the task identifier and configuration record to find potentially incorrect recommendations. A model deployment name alone may not describe every relevant component of the release.

Then verify the business impact. Determine whether the recommendation was merely generated, accepted by a reviewer, or acted upon. Those are different incident scopes and may require different communication and correction.

Finally, reconcile unfinished work. A rollback can restore a prior configuration, but it cannot determine whether a previous tool call completed. Use authoritative records to resolve uncertain outcomes.

Turn the incident into a release test

Build a regression fixture from the failure with permitted data. Include the confusing variants and the correct procedure. Run it against the proposed correction and the baseline.

Add neighbouring cases rather than a single memorized example. If a metadata filter fixes one product but excludes valid sources for another, the correction is incomplete.

The July hosted-agent GA announcement is useful context for managed operations, while the earlier Agent Framework release description explains orchestration foundations. Neither replaces an application-specific incident test. The release decision should use evidence from the actual support workflow.

Connect observability to cost and accountability

Calculate evaluation and telemetry cost alongside model and runtime spend. Sample ordinary traffic where appropriate, while deliberately retaining evidence for important failures under the chosen policy.

Report time to detect a quality regression, time to identify its cause, unresolved affected tasks, and recurrence after correction. Pair those measures with accepted business outcomes and human correction effort.

Assign an incident owner who can coordinate data, application, model, and platform teams. The purpose of a trace is to shorten the path from a bad outcome to a verified correction. A richer dashboard is useful only when it helps that process.

For the first rollout, choose one task family and walk through a controlled wrong-answer incident from detection to regression test. If the team can explain the failure and prove the correction, observability is doing practical work.

References

Microsoft: Agent Service evaluation and monitoring, March 16, 2026

Microsoft: Observability and ROI, June 3, 2026

Yao and colleagues: τ-bench, 2024

Microsoft: Hosted-agent GA announcement, July 9, 2026

Microsoft: Agent Framework architecture, February 19, 2026

An enterprise may know who owns its customer database and still struggle to name the owner of an agent that reads it, calls a pricing service, and prepares a customer offer. The agent crosses several teams’ boundaries in one task. Governance has to connect those responsibilities.

The first control I would establish is a clear answer to three questions: who owns the outcome, whose authority the agent uses, and who can stop its actions. A product inventory helps, but an entry in a catalogue is not an operating agreement.

Production hosting changes the operating obligation

Microsoft’s July 9 Foundry announcement identifies hosted agents as generally available alongside new models and deployment capabilities. Microsoft also reports that more than 100,000 organizations are building on Foundry. That is a vendor-reported adoption figure, not a measure of successful deployments or proof of governance maturity.

A managed hosting milestone can remove infrastructure work. It does not remove the application owner’s responsibility for incorrect quotes, unauthorized access, or unfinished transactions. Review the agent as a business application with delegated authority.

The April developer announcement and June hosted-agent update provide background on runtimes, identity, and controls. Read feature-specific maturity and deployment requirements rather than carrying a general GA label across every related capability.

Define the business action before the tool

Consider a hypothetical sales-support agent that assembles a renewal proposal. Its read operations retrieve contract terms and approved price information. Its write operations might save an internal draft or submit an offer for review. Sending the offer to the customer is a separate business action.

These operations have different consequences and should have separate authorization rules. A broad tool called “manage renewal” makes it difficult to distinguish harmless preparation from an external commitment.

Create narrow operations with explicit targets and validated arguments. The application should resolve the customer account from authenticated context and verified records. An instruction embedded in a document must not select a different tenant, customer, recipient, or authority.

Establish an owner for each control

The business owner defines permitted outcomes, escalation rules, and materiality. The application owner implements transaction validation and recovery. Identity teams establish access and credential lifecycle. Platform teams own deployment, monitoring, and configuration distribution.

Security teams should assess abuse paths, but they should not become the default owner of every unresolved application decision. A policy requiring approval is ineffective unless an application team implements it and a business team supplies authorized reviewers.

Record who can change prompts, tools, model deployments, and policy configuration. Treat those changes as application releases. A small configuration update can increase authority without changing the repository’s main code path.

Use adversarial research to improve test design

The 2024 AgentDojo study created 97 realistic tasks and 629 security test cases for agents operating on untrusted tool data. Those counts describe benchmark coverage, not the probability that a Foundry agent will be compromised. The study’s value here is its stateful testing approach.

A renewal agent should encounter a supplier document containing instructions to send information elsewhere, a tool response claiming that approval is unnecessary, and a retrieved note that conflicts with current policy. Score whether the agent completes the legitimate task and whether the application blocks prohibited actions.

Microsoft’s AI Red Teaming Agent announcement describes preview testing for risks including sensitive-data leakage and prohibited actions. Treat such tools as part of a wider evaluation programme. Confirm that the chosen tests reflect the actual tools and permissions of the application.

Keep policy enforcement outside model persuasion

Prompts should explain expected behaviour. Deterministic controls should enforce resource scope, recipient restrictions, financial bounds, and required approvals. A model-generated justification must not grant an exception.

Where a tool accepts a record identifier, validate that the caller may act on that record. Where it accepts an external destination, restrict that destination according to application policy. These checks remain necessary even when the agent usually chooses correct parameters.

Do not give every tool the same credential. Narrow identities make the effect of a compromised component easier to contain and investigate. Confirm the selected hosting path’s actual identity behaviour before assuming a deployment supplies the intended separation.

Plan for changes in authority

An agent that begins with read-only access can accumulate privileges through normal product development. Review new tools against the original task boundary. Adding an API may create a new disclosure route or business commitment.

Maintain an inventory of active agent versions, tool definitions, policy versions, and approved deployment destinations. Include dependencies outside Foundry where they participate in the workflow. The governance boundary should follow the business action across services.

Test removal of access as well as addition. A revoked role or disabled connection should lead to a clear failure or escalation. It should not cause the agent to search for a broader credential or alternate route.

Make stopping and recovery operational

Define how to stop new work, prevent pending writes, and reconcile actions already submitted. A deployment rollback does not reverse an offer that was sent or a customer record that was changed.

The runbook should identify the authoritative transaction log and the operator who can determine what happened. Preserve operation identifiers so incident responders can distinguish an attempted action from a completed one.

Exercise this process with a controlled failure. The test should show how unfinished work is located, how the customer impact is assessed, and how service resumes. Record the evidence rather than accepting a diagram as proof.

Give leaders a useful governance report

Report agents by business owner and permitted action scope. Include failed authorization tests, overdue remediation, unresolved transaction outcomes, and changes that increased autonomy. Separate these from general traffic and token usage.

For the renewal example, pair technical measures with incorrectly scoped proposals, review corrections, and confirmed customer impact. A dashboard with many successful model responses can still conceal a serious authority problem.

Begin expansion only when the current boundary is understood and tested. Foundry can provide useful hosting and governance building blocks. Enterprise control comes from connecting those capabilities to named owners, enforceable authority, and evidence about the resulting business actions.

References

Microsoft: July 9, 2026 Foundry announcement

Microsoft: Developer journey and identity foundations, April 22, 2026

Microsoft: Hosted-agent controls, June 2, 2026

Microsoft: AI Red Teaming Agent, November 2025

Debenedetti and colleagues: AgentDojo, 2024