Skip to content

THR902 Theater Thu 19 Nov 17:30 PST

Take control of runaway AI costs with AI Gateway

Mike Budzynski

Summary

From the recording

Mike Budzinski, a Microsoft Principal Product Manager, presented Azure AI Gateway, which he said became generally available two days before the session. It sits between AI clients and models in Foundry, AWS Bedrock, OpenAI, Anthropic or Google Cloud. The demo covered OpenTelemetry cost monitoring, Azure RBAC model access, a cost limit policy, aliases with priorities, weights and session affinity, and auto model selection. He said it is free to start with. Caveat: cost figures are estimates from observed tokens and public pricing, and the demo used fictional models.

Session description

Microsoft's catalogue description

As AI usage grows across applications and agents, costs can become difficult to predict and manage. Learn how AI Gateway helps platform teams understand and control AI consumption using centralized visibility, token quotas, rate limits, semantic caching, and intelligent routing. See how to apply FinOps controls directly in the AI runtime to improve efficiency, reduce waste, and support AI adoption at scale.

← All sessions