BRK251 Breakout Wed 18 Nov 15:00 PST
Agents in production: Observe, evaluate, and optimize with Foundry
Han Che, Divya Chirasani, Deepraj Khot
Summary
From the recording
Microsoft said Foundry's insights from production traces are generally available, while three new Agent Optimizer strategies and human annotation are in public preview. A fleet view for monitoring agents is in preview. Microsoft cited a Gartner study expecting 40% of production agents to be disabled by end of 2027. In a demo, an agent scored 67% on an offline dataset, then fell below the demo's 65% target on live traffic. Caveat: based on auto-generated captions; no pricing, regions or GA dates were given for preview items, and survey sources were unnamed.
Session description
Microsoft's catalogue description
Getting an agent into production is only the beginning. Learn how Microsoft Foundry helps teams observe agent behavior, evaluate quality, uncover unexpected issues, and optimize performance over time. Through practical guidance, demos, and a customer story from Tata Consultancy Services, explore how tracing, evaluations, insights, human feedback, and optimization come together in a repeatable continuous improvement loop. You’ll also see how to monitor quality, reliability, latency, safety, and cost across your agent fleet.
Demo timestamps
Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.
- Skip to 0:07:18
Agent traces, conversation view and human feedback
The traces tab shows individual requests or a conversation trajectory, including duration, tokens, user question and metadata. Traces can be annotated with human feedback.
- Skip to 0:11:22
Auto-generated quality rubric
For an investment agent that reads files and cites sources, a quality rubric is auto-generated covering retrieval, citations and conflict handling.
- Skip to 0:13:29
Offline evaluation on a golden dataset
A one-time evaluation runs on a golden dataset, called offline because it is not live traffic. A UI glitch closes the run, so a completed example run is shown instead.
- Skip to 0:19:48
Live insights for a portfolio-review agent
For a portfolio review agent that should summarise evidence but not answer confidently when evidence is thin, a judge model is selected, scheduled analysis is enabled, and existing scans are reviewed on the insights page.
- Skip to 0:26:42
Fix Insight into Agent Optimizer
A Fix Insight control creates an optimisation run with an agent version, default optimisation and evaluation models, and a maximum number of candidates. Jobs are said to take minutes to hours.
- Skip to 0:33:06
Project-level monitor page
The monitor page lists active agents and Foundry models for a project, with usage, tokens, cost, policy violations, errors and latency.