The monitoring dashboard is green. The agent’s requests completed, latency is within target, and no service returned an error. Yet a support specialist reports that the assistant recommended the wrong maintenance procedure.
This is an application incident even if every API returned success. Investigating it requires evidence about the retrieved source, selected tool, model configuration, and resulting recommendation. Infrastructure health is one part of that evidence.
Trace the business task across components
Microsoft’s March Agent Service announcement describes evaluation and monitoring capabilities as part of the platform. The June observability announcement adds a more detailed map of the development loop, including preview extensions for multi-turn evaluation, trace replay, and turning traces into datasets.
Do not assume that a foundational GA capability gives every new extension the same status. Identify which tracing, evaluation, and retention surfaces the application uses and verify them separately.
My recommendation is to start with an application task identifier. Propagate it through orchestration, retrieval, model requests, tool calls, and review outcomes. That makes it possible to ask what happened to one customer request across several systems.
A successful response is not a successful task
The 2024 τ-bench research found less than 50% task success for its tested function-calling agents and retail pass^8 below 25%. Those historical results are not a scorecard for current Foundry deployments. They demonstrate why repeated task-level checks can reveal weaknesses that a plausible conversation hides.
For a support agent, a success label should depend on the correct recommendation and permitted final state. If the agent cites an obsolete procedure, record that as a quality failure even when retrieval and generation completed normally.
Separate task success, policy compliance, and service availability. Combining them into one green indicator makes operational decisions harder. Different failure classes require different owners and remedies.
Reconstruct an illustrative incident
Imagine a hypothetical industrial support assistant that advises replacing a component. A reviewer later discovers that the equipment variant uses a different procedure. The trace shows the assistant retrieved a manual for the correct product family but the wrong variant.
That evidence points first to retrieval and metadata filtering. Changing the model’s general instructions might suppress the symptom without fixing the source selection. The investigation should establish where the wrong variant entered the workflow.
Retain the source document identifier, revision, retrieval filter, and tool arguments where permitted. Store the model and prompt configuration alongside the task. Without those details, a later replay may use changed sources and produce a misleadingly correct answer.
Collect evidence deliberately
Content capture can expose customer information, proprietary documents, and credentials accidentally returned by tools. Decide what the investigation genuinely needs before enabling broad logging.
Use identifiers, timing, configuration versions, and outcome labels as the default operational layer. Capture sensitive content only through a defined process with access controls and retention appropriate to its purpose.
Test the telemetry pipeline with representative sensitive inputs. Verify redaction where intended and check error paths, because exceptions often contain more information than normal logs. A configuration that protects successful requests may still leak content when a dependency fails.
Monitor quality by meaningful slices
Aggregate accuracy can improve while a critical equipment class becomes worse. Group results by task type, language, source family, model configuration, and relevant tool path. Keep small samples visible rather than reporting unstable percentages with false precision.
Use human review for cases where domain judgement is essential. Calibrate any model-based evaluator against that review. Track disagreement so an evaluator change does not silently redefine what the dashboard calls success.
Compare the current release with a stable baseline. Alert on a meaningful change in important task outcomes, then inspect whether the cause is a new model, revised source, changed tool, or traffic shift.
Design an incident response sequence
First contain the consequence. Pause a problematic write path or direct affected requests to review while preserving safe read-only assistance where appropriate.
Next identify the affected versions and tasks. Use the task identifier and configuration record to find potentially incorrect recommendations. A model deployment name alone may not describe every relevant component of the release.
Then verify the business impact. Determine whether the recommendation was merely generated, accepted by a reviewer, or acted upon. Those are different incident scopes and may require different communication and correction.
Finally, reconcile unfinished work. A rollback can restore a prior configuration, but it cannot determine whether a previous tool call completed. Use authoritative records to resolve uncertain outcomes.
Turn the incident into a release test
Build a regression fixture from the failure with permitted data. Include the confusing variants and the correct procedure. Run it against the proposed correction and the baseline.
Add neighbouring cases rather than a single memorized example. If a metadata filter fixes one product but excludes valid sources for another, the correction is incomplete.
The July hosted-agent GA announcement is useful context for managed operations, while the earlier Agent Framework release description explains orchestration foundations. Neither replaces an application-specific incident test. The release decision should use evidence from the actual support workflow.
Connect observability to cost and accountability
Calculate evaluation and telemetry cost alongside model and runtime spend. Sample ordinary traffic where appropriate, while deliberately retaining evidence for important failures under the chosen policy.
Report time to detect a quality regression, time to identify its cause, unresolved affected tasks, and recurrence after correction. Pair those measures with accepted business outcomes and human correction effort.
Assign an incident owner who can coordinate data, application, model, and platform teams. The purpose of a trace is to shorten the path from a bad outcome to a verified correction. A richer dashboard is useful only when it helps that process.
For the first rollout, choose one task family and walk through a controlled wrong-answer incident from detection to regression test. If the team can explain the failure and prove the correction, observability is doing practical work.
References
Microsoft: Agent Service evaluation and monitoring, March 16, 2026
Microsoft: Observability and ROI, June 3, 2026
Yao and colleagues: τ-bench, 2024
