BRK441 Breakout Tue 17 Nov 16:30 PST
Inside Azure Maia: Purpose-built for AI inference at scale
Vikram Vemulapalli, Andrew Wall
Summary
From the recording
Microsoft said Maia 200 is live in data centres running production workloads for its first-party services, delivering about 40 to 70% more tokens per megawatt. Maia 300, announced a few weeks earlier, uses a multi-chiplet design with 8 HBM stacks and 18 TB/s of HBM bandwidth. Racks move from 48 accelerators to 96, giving 5x aggregate rack-level HBM bandwidth, and up to 12,000 Maia 300s can connect. Microsoft expects 7x rack-scale token throughput versus Maia 200. Maia 300 is expected next year. Caveat: the baseline for the 7x figure is unclear; customer access, pricing and regions were not stated.
Session description
Microsoft's catalogue description
Building world-class AI infrastructure requires more than a great chip. Learn how Azure Maia combines custom silicon, systems, networking, and software into a unified platform optimized for AI inference and see how end-to-end co-design translates into better customer outcomes.