ODSP2110 Pre-recorded On demand
Frontier-driven AI: Evaluating LLMs at scale
Robert Madison, Zachary Richardson
Summary
From Microsoft's AI summary
This session presents a Microsoft Foundry model evaluation promotion workflow for regulated settings such as healthcare. It proposes a versioned golden set of about 60 offline cases scoring quality, safety and FinOps, including cost per hard case won, plus a live subset of roughly eight cases before human approval. The demo uses a GitHub repository and GitHub Actions pipelines, with a Llama 4 baseline, a GPT-5.4 judge, and Maverick and DeepSeek v3.2 candidates. It ends with a report in VS Code and JSON artefacts. No availability or pricing was stated.
Session description
Microsoft's catalogue description
Enterprises are adopting foundation models at an unprecedented pace, creating new challenges around governance, risk, cost, and model quality. This demo shows an automated evaluation and promotion framework built on Microsoft Foundry, using benchmark datasets, safety testing, and cost analysis to assess candidate models. Attendees will see how organizations can establish trusted baselines, automate decision-making, and deploy AI solutions with governance and operational controls at scale.