Skip to content

ODSP2110 Pre-recorded On demand

Frontier-driven AI: Evaluating LLMs at scale

Robert Madison, Zachary Richardson

Summary

From Microsoft's AI summary

This session presents a Microsoft Foundry model evaluation promotion workflow for regulated settings such as healthcare. It proposes a versioned golden set of about 60 offline cases scoring quality, safety and FinOps, including cost per hard case won, plus a live subset of roughly eight cases before human approval. The demo uses a GitHub repository and GitHub Actions pipelines, with a Llama 4 baseline, a GPT-5.4 judge, and Maverick and DeepSeek v3.2 candidates. It ends with a report in VS Code and JSON artefacts. No availability or pricing was stated.

Session description

Microsoft's catalogue description

Enterprises are adopting foundation models at an unprecedented pace, creating new challenges around governance, risk, cost, and model quality. This demo shows an automated evaluation and promotion framework built on Microsoft Foundry, using benchmark datasets, safety testing, and cost analysis to assess candidate models. Attendees will see how organizations can establish trusted baselines, automate decision-making, and deploy AI solutions with governance and operational controls at scale.

← All sessions