THR902 Theater Thu 19 Nov 17:30 PST
Take control of runaway AI costs with AI Gateway
Mike Budzynski
Summary
From the recording
Mike Budzinski, a Microsoft Principal Product Manager, presented Azure AI Gateway, which he said became generally available two days before the session. It sits between AI clients and models in Foundry, AWS Bedrock, OpenAI, Anthropic or Google Cloud. The demo covered OpenTelemetry cost monitoring, Azure RBAC model access, a cost limit policy, aliases with priorities, weights and session affinity, and auto model selection. He said it is free to start with. Caveat: cost figures are estimates from observed tokens and public pricing, and the demo used fictional models.
Session description
Microsoft's catalogue description
As AI usage grows across applications and agents, costs can become difficult to predict and manage. Learn how AI Gateway helps platform teams understand and control AI consumption using centralized visibility, token quotas, rate limits, semantic caching, and intelligent routing. See how to apply FinOps controls directly in the AI runtime to improve efficiency, reduce waste, and support AI adoption at scale.