BRK233 Breakout Wed 18 Nov 15:00 PST
Beyond the Benchmark: Evaluating Copilot and AI Tools
Bas Brekelmans, Amber Waisanen
Summary
From the recording
Microsoft speakers set out a four-part framework for evaluating AI tools: accuracy, trust, speed and cost, judged together rather than by public benchmarks or hours saved. Copilot Chat was described as included in the licence today, while Copilot Cowork is usage-based. Bas Brekelmans said a query-processing change made the product 24% cheaper over the last month, and that KV caching can give a 90% cost reduction on reused tokens; both are speaker claims. An admin cost view with use-case reporting was described as new at Ignite, starting with Cowork, with other experiences coming. No dates or SKUs were stated.
Session description
Microsoft's catalogue description
How do you know which AI delivers the most value for your organization? In this session, you'll learn a repeatable framework and methodology for evaluating Microsoft Copilot, Copilot Cowork and other common AI tools against real work scenarios. We'll walk through how to measure accuracy, trust, speed and cost, so you leave equipped to test your own scenarios, judge where each tool delivers value, and share feedback that helps shape better Copilot experiences.
Demo timestamps
Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.
- Skip to 0:07:26
Live Copilot Chat demo for decision-making and document creation
In a pharmaceutical launch planning scenario at the fictional company Caldova, Copilot Chat analyzes meeting attendees and workshop data, summarizes stakeholder positions, and generates an executive briefing that keeps inherited confidentiality labels.
- Skip to 0:12:39
Copilot Cowork with model choice and workflow automation
Cowork gives more control over which models are used. A customer escalation scenario shows it investigating an outage by searching ERP, CRM, finance records and enterprise files.