Skip to content

BRK233 Breakout Wed 18 Nov 15:00 PST

Beyond the Benchmark: Evaluating Copilot and AI Tools

Bas Brekelmans, Amber Waisanen

Summary

From the recording

Microsoft speakers set out a four-part framework for evaluating AI tools: accuracy, trust, speed and cost, judged together rather than by public benchmarks or hours saved. Copilot Chat was described as included in the licence today, while Copilot Cowork is usage-based. Bas Brekelmans said a query-processing change made the product 24% cheaper over the last month, and that KV caching can give a 90% cost reduction on reused tokens; both are speaker claims. An admin cost view with use-case reporting was described as new at Ignite, starting with Cowork, with other experiences coming. No dates or SKUs were stated.

Session description

Microsoft's catalogue description

How do you know which AI delivers the most value for your organization? In this session, you'll learn a repeatable framework and methodology for evaluating Microsoft Copilot, Copilot Cowork and other common AI tools against real work scenarios. We'll walk through how to measure accuracy, trust, speed and cost, so you leave equipped to test your own scenarios, judge where each tool delivers value, and share feedback that helps shape better Copilot experiences.

Demo timestamps

Microsoft's player does not support links to a specific time, so open the recording and skip to the time shown.

  1. Skip to 0:07:26

    Live Copilot Chat demo for decision-making and document creation

    In a pharmaceutical launch planning scenario at the fictional company Caldova, Copilot Chat analyzes meeting attendees and workshop data, summarizes stakeholder positions, and generates an executive briefing that keeps inherited confidentiality labels.

  2. Skip to 0:12:39

    Copilot Cowork with model choice and workflow automation

    Cowork gives more control over which models are used. A customer escalation scenario shows it investigating an outage by searching ERP, CRM, finance records and enterprise files.

Watch on-demand (opens in a new tab)

← All sessions