Skip to content
Status: Generally available Managed and programmed training in preview

Foundry continuous improvement: multi-turn evaluation, production insights and Agent Optimizer

Foundry continuous improvement capabilities include multi-turn evaluation, production insights and Agent Optimizer, which are available now. Managed and programmed training are in preview.

Sources

BRK249 Breakout Wed 18 Nov 11:30 PST

From traces to better agents: The Foundry improvement loop

Avinav Jami, Charles Kim, Takashi Okamoto, Yuji Shono

From the recording

Microsoft said Agent Insights is generally available, as is tracing for agents built outside Foundry, including OpenTelemetry traces. Trace annotation, rubric evaluators and golden-dataset evaluation are in preview, as are the Insights-to-Agent Optimizer handoff, skill and prompt optimisation, cost optimisation, model router and training. In a demo, one optimisation turn raised an evaluation score from 36% to 89%. Caveat: captions are auto-generated, the demo result is a single case, and no pricing was stated.

Demo timestamps

Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.

  1. Skip to 0:04:39

    Long-running Foundry hosted agent over Fabric IQ

    A hosted agent, connected through Foundry's toolbox to several programmes and departments over a Fabric connection, is already running a long task while the presenter waits on a slow response.

  2. Skip to 0:09:24

    Trace view and optimiser run on the agent

    The agent's recent conversations are found in the trace view. An optimiser run finds the agent was not consulting every department. Evaluation score goes from 36% to a single prompt producing a cross-department programme update.

  3. Skip to 0:26:52

    PMO agent architecture walkthrough

    Walkthrough of a PMO agent that monitors project status from Azure DevOps and Teams data retrieved via Work IQ, drafts reports and follow-ups, and waits for a project manager's approval. Foundry is the improvement loop around it.

Watch on-demand (opens in a new tab)

BRK251 Breakout Wed 18 Nov 15:00 PST

Agents in production: Observe, evaluate, and optimize with Foundry

Han Che, Divya Chirasani, Deepraj Khot

From the recording

Microsoft said Foundry's insights from production traces are generally available, while three new Agent Optimizer strategies and human annotation are in public preview. A fleet view for monitoring agents is in preview. Microsoft cited a Gartner study expecting 40% of production agents to be disabled by end of 2027. In a demo, an agent scored 67% on an offline dataset, then fell below the demo's 65% target on live traffic. Caveat: based on auto-generated captions; no pricing, regions or GA dates were given for preview items, and survey sources were unnamed.

Demo timestamps

Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.

  1. Skip to 0:07:18

    Agent traces, conversation view and human feedback

    The traces tab shows individual requests or a conversation trajectory, including duration, tokens, user question and metadata. Traces can be annotated with human feedback.

  2. Skip to 0:11:22

    Auto-generated quality rubric

    For an investment agent that reads files and cites sources, a quality rubric is auto-generated covering retrieval, citations and conflict handling.

  3. Skip to 0:13:29

    Offline evaluation on a golden dataset

    A one-time evaluation runs on a golden dataset, called offline because it is not live traffic. A UI glitch closes the run, so a completed example run is shown instead.

  4. Skip to 0:19:48

    Live insights for a portfolio-review agent

    For a portfolio review agent that should summarise evidence but not answer confidently when evidence is thin, a judge model is selected, scheduled analysis is enabled, and existing scans are reviewed on the insights page.

  5. Skip to 0:26:42

    Fix Insight into Agent Optimizer

    A Fix Insight control creates an optimisation run with an agent version, default optimisation and evaluation models, and a maximum number of candidates. Jobs are said to take minutes to hours.

  6. Skip to 0:33:06

    Project-level monitor page

    The monitor page lists active agents and Foundry models for a project, with usage, tokens, cost, policy violations, errors and latency.

Watch on-demand (opens in a new tab)

More in this area

← All announcements