BRK249 Breakout Wed 18 Nov 11:30 PST
From traces to better agents: The Foundry improvement loop
Avinav Jami, Charles Kim, Takashi Okamoto, Yuji Shono
Summary
From the recording
Microsoft said Agent Insights is generally available, as is tracing for agents built outside Foundry, including OpenTelemetry traces. Trace annotation, rubric evaluators and golden-dataset evaluation are in preview, as are the Insights-to-Agent Optimizer handoff, skill and prompt optimisation, cost optimisation, model router and training. In a demo, one optimisation turn raised an evaluation score from 36% to 89%. Caveat: captions are auto-generated, the demo result is a single case, and no pricing was stated.
Session description
Microsoft's catalogue description
Building an agent is the start. Improving it in production is the real work. See how Microsoft Foundry connects traces, evaluations, and optimization to help teams find high-impact issues, test improvements, and re-measure results. Learn a repeatable loop to improve quality, manage cost, and ship better agents while developers stay in control.
Demo timestamps
Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.
- Skip to 0:04:39
Long-running Foundry hosted agent over Fabric IQ
A hosted agent, connected through Foundry's toolbox to several programmes and departments over a Fabric connection, is already running a long task while the presenter waits on a slow response.
- Skip to 0:09:24
Trace view and optimiser run on the agent
The agent's recent conversations are found in the trace view. An optimiser run finds the agent was not consulting every department. Evaluation score goes from 36% to a single prompt producing a cross-department programme update.
- Skip to 0:26:52
PMO agent architecture walkthrough
Walkthrough of a PMO agent that monitors project status from Azure DevOps and Teams data retrieved via Work IQ, drafts reports and follow-ups, and waits for a project manager's approval. Foundry is the improvement loop around it.