Skip to content

THR862 Theater Thu 19 Nov 16:55 PST

Prompt caching at scale: Build faster, more efficient AI applications

Saloni Sonpal, Taylor Wrobel

Session description

Microsoft's catalogue description

Every AI request pays for a prefill phase before it generates a single token, and most production apps recompute the same system prompts, tool definitions, and retrieved context on every turn. Prompt caching removes that waste.In this session, learn how prompt caching reduce AI application cost and latency while increasing throughput, and how to pick the right Azure OpenAI caching strategy for your workload, from automatic to fully managed.

← All sessions