Skip to content

ODSP2235 Pre-recorded On demand

Build an enterprise LLM platform with vLLM, Ray, and Anyscale

Daniel Arrizza

Summary

From Microsoft's AI summary

Daniel Arrizza of Anyscale demonstrates hosting an open coding model inside a customer's Azure subscription using vLLM, Ray Serve and Anyscale on AKS. The control plane manages scheduling while compute runs in the customer's VNet, with weights, prompts and code staying in their tenant. He deploys Qwen 3.6 27B with FP8 quantisation on A100 node pools, tests the APIs, and connects GitHub Copilot to generate JavaScript. He reviews dashboard metrics and says one GPU can support several dozen developer agent sessions. No availability or pricing was stated.

Session description

Microsoft's catalogue description

Whether you want to cut costs or gain control over your AI, open models offer a path forward. The hard parts are running LLMs at scale and building a platform that goes beyond inference, all securely within your environment. Ray is the open-source framework AI natives use to scale the full AI lifecycle, and Anyscale with its Azure-native service makes it enterprise-ready. In this session, we’ll deploy a coding agent on open models using vLLM and Ray Serve, all running in your own account.

← All sessions