# AI Gateway An AI gateway is a proxy layer between applications and AI model providers. It provides a unified API across multiple providers ([[Anthropic]], [[OpenAI]], [[Mistral AI]], [[Deepseek]], etc.), adding cross-cutting concerns like authentication, rate limiting, cost tracking, caching, logging, and [[Model routing]]. Key capabilities: - **Provider abstraction**: switch between models and providers without changing application code - **Fallback and retry**: automatically route to a backup provider if the primary is down or rate-limited - **Cost management**: set budgets, track spend per team/project/user - **[[Prompt caching]]**: cache repeated prompts to reduce latency and cost - **Observability**: log all requests/responses for [[AI Observability]] and debugging - **Guardrails**: apply [[AI Guardrails]] (content filtering, PII detection) at the gateway level [[OpenRouter]] is a well-known AI gateway that aggregates dozens of models behind a single API, with automatic routing and pricing optimization. The category splits along two axes: who operates it, and whether it sits in your data path. **Managed gateways** (someone else runs it, your traffic crosses their network): - [[Vercel AI Gateway]], [[Cloudflare AI Gateway]], [[Portkey]], [[Requesty]], [[Helicone]] - [[OpenRouter]] and the cost-focused routers built on top of it: [[Ramp Router]], [[Cheaper Inference]] **Self-hosted gateways** (you run it, traffic stays inside your infrastructure): - [[LiteLLM]] for a single team, [[Kong AI Gateway]] if you already run Kong, [[Envoy AI Gateway]] if your platform is Envoy and [[Kubernetes]] **Routing without a proxy hop**: [[Not Diamond]] decides which model should answer while staying out of the request path itself. Fewer custodians, same routing benefit. One thing to watch across the whole category: per-token markup has collapsed to roughly zero at most of the managed players, so pricing is no longer the differentiator. What is left to compare is deployment model, failover behaviour, and what each vendor stores. For teams building [[AI Agents]] and [[Agentic Engineering]] workflows, a gateway reduces vendor lock-in, simplifies cost control, and provides a single point for observability and security. It's the AI equivalent of an API gateway in microservice architectures. ## References - ## Related - [[Model routing]] - [[OpenRouter]] - [[Vercel AI Gateway]] - [[Cloudflare AI Gateway]] - [[LiteLLM]] - [[Portkey]] - [[Helicone]] - [[Requesty]] - [[Kong AI Gateway]] - [[Envoy AI Gateway]] - [[Not Diamond]] - [[Ramp Router]] - [[Cheaper Inference]] - [[AI Observability]] - [[AI Guardrails]] - [[Large Language Models (LLMs)]] - [[Anthropic]] - [[OpenAI]] - [[AI Agents]]