Foundry continuous improvement: multi-turn evaluation, production insights and Agent Optimizer
Foundry continuous improvement capabilities include multi-turn evaluation, production insights and Agent Optimizer, which are available now. Managed and programmed training are in preview.
Sources
Related sessions
BRK249 Breakout Wed 18 Nov 11:30 PST
From traces to better agents: The Foundry improvement loop
Avinav Jami, Charles Kim, Takashi Okamoto, Yuji Shono
From the recording
Microsoft said Agent Insights is generally available, as is tracing for agents built outside Foundry, including OpenTelemetry traces. Trace annotation, rubric evaluators and golden-dataset evaluation are in preview, as are the Insights-to-Agent Optimizer handoff, skill and prompt optimisation, cost optimisation, model router and training. In a demo, one optimisation turn raised an evaluation score from 36% to 89%. Caveat: captions are auto-generated, the demo result is a single case, and no pricing was stated.
Demo timestamps
Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.
- Skip to 0:04:39
Long-running Foundry hosted agent over Fabric IQ
A hosted agent, connected through Foundry's toolbox to several programmes and departments over a Fabric connection, is already running a long task while the presenter waits on a slow response.
- Skip to 0:09:24
Trace view and optimiser run on the agent
The agent's recent conversations are found in the trace view. An optimiser run finds the agent was not consulting every department. Evaluation score goes from 36% to a single prompt producing a cross-department programme update.
- Skip to 0:26:52
PMO agent architecture walkthrough
Walkthrough of a PMO agent that monitors project status from Azure DevOps and Teams data retrieved via Work IQ, drafts reports and follow-ups, and waits for a project manager's approval. Foundry is the improvement loop around it.
BRK251 Breakout Wed 18 Nov 15:00 PST
Agents in production: Observe, evaluate, and optimize with Foundry
Han Che, Divya Chirasani, Deepraj Khot
From the recording
Microsoft said Foundry's insights from production traces are generally available, while three new Agent Optimizer strategies and human annotation are in public preview. A fleet view for monitoring agents is in preview. Microsoft cited a Gartner study expecting 40% of production agents to be disabled by end of 2027. In a demo, an agent scored 67% on an offline dataset, then fell below the demo's 65% target on live traffic. Caveat: based on auto-generated captions; no pricing, regions or GA dates were given for preview items, and survey sources were unnamed.
Demo timestamps
Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.
- Skip to 0:07:18
Agent traces, conversation view and human feedback
The traces tab shows individual requests or a conversation trajectory, including duration, tokens, user question and metadata. Traces can be annotated with human feedback.
- Skip to 0:11:22
Auto-generated quality rubric
For an investment agent that reads files and cites sources, a quality rubric is auto-generated covering retrieval, citations and conflict handling.
- Skip to 0:13:29
Offline evaluation on a golden dataset
A one-time evaluation runs on a golden dataset, called offline because it is not live traffic. A UI glitch closes the run, so a completed example run is shown instead.
- Skip to 0:19:48
Live insights for a portfolio-review agent
For a portfolio review agent that should summarise evidence but not answer confidently when evidence is thin, a judge model is selected, scheduled analysis is enabled, and existing scans are reviewed on the insights page.
- Skip to 0:26:42
Fix Insight into Agent Optimizer
A Fix Insight control creates an optimisation run with an agent version, default optimisation and evaluation models, and a maximum number of candidates. Jobs are said to take minutes to hours.
- Skip to 0:33:06
Project-level monitor page
The monitor page lists active agents and Foundry models for a project, with usage, tokens, cost, policy violations, errors and latency.
THR860 Theater Thu 19 Nov 14:15 PST
Make every agent better: The continuous optimization loop
Vivek Bhadauria, Bhuwan Sahni
More in this area
- Status: Generally available AI Gateway for unified AI management and policy
- Status: Generally available Azure Catalyst: agent-led discovery, business case, landing zones and migration
- Status: Generally available Azure Chaos Studio Workspaces and AKS extension
- Status: Generally available Azure Cobalt 200 VMs