# Kong AI Gateway Kong AI Gateway is a set of AI plugins layered on top of Kong Gateway, the long-standing API gateway. The pitch is simple: you already run an API gateway, so put your LLM traffic through the same governance layer instead of standing up a second one. The technical difference from plain proxying is that Kong reads the prompts. Put an LLM API behind a normal gateway and you only see HTTP. The AI plugins understand the payload, which is what makes semantic caching, prompt guarding and PII redaction possible at all. ## The plugin set - **AI Proxy** and **AI Proxy Advanced**: one standardized interface across [[OpenAI]], [[Anthropic]], Azure AI and others - **Semantic caching and semantic routing**: cache and route on meaning rather than exact string matches - **Load balancing**: round-robin, consistent hashing, or latency-based - **AI Prompt Guard**, prompt and response guards, plus **PII sanitizer** covering 20 categories across 9 languages - **AI RAG Injector**, prompt decorator and prompt compressor - **Token-based rate limiting**, which is the unit that actually matters for LLM spend ## Who it is for Enterprises already standardized on Kong. If that is not you, the operational weight is hard to justify against [[LiteLLM]] or a managed option. If it is you, this is the least disruptive path, and it runs self-hosted for full control, or through Konnect, hybrid, DB-less and Kubernetes deployments. Token-based rate limiting deserves a special mention. Request-count limits are close to meaningless when one agent turn can burn a hundred times the tokens of another. ## References - AI Gateway documentation: https://developer.konghq.com/ai-gateway/ - AI Proxy plugin: https://developer.konghq.com/plugins/ai-proxy/ - Gateway AI documentation: https://docs.konghq.com/gateway/latest/ai-gateway/ - Repository: https://github.com/Kong/kong ## Related - [[AI Gateway]] - [[Model routing]] - [[AI Guardrails]] - [[Semantic Search]] - [[LiteLLM]] - [[Kubernetes]] - [[OpenAI]] - [[Anthropic]]