Bringing AI-Aware Traffic Management to Istio: Gateway API Inference Extension Support

A smarter, dynamic way to optimize AI traffic routing based on real-time metrics and the unique characteristics of inference workloads.

Jul 28, 2025 | By Lior Lieberman - Google, Keith Mattix - Microsoft, Aslak Knutsen - Red Hat

The world of AI inference on Kubernetes presents unique challenges that traditional traffic-routing architectures weren’t designed to handle. While Istio has long excelled at managing microservice traffic with sophisticated load balancing, security, and observability features, the demands of Large Language Model (LLM) workloads require specialized functionality.

That’s why we’re excited to announce Istio’s support for the Gateway API Inference Extension, bringing intelligent, model-aware and LoRA-aware routing to Istio.

Why AI Workloads Need Special Treatment

Traditional web services typically handle quick, stateless requests measured in milliseconds. AI inference workloads operate in a completely different paradigm that challenges conventional load balancing approaches in several fundamental ways.

The Scale and Duration Challenge

Unlike typical API responses that complete in milliseconds, AI inference requests often take significantly longer to process - sometimes several seconds or even minutes. This dramatic difference in processing time means that routing decisions have far more impact than in traditional web services. A single poorly-routed request can tie up expensive GPU resources for extended periods, creating cascading effects across the entire system.

The payload characteristics are equally challenging. AI inference requests frequently involve substantially larger payloads, especially when dealing with Retrieval-Augmented Generation (RAG) systems, multi-turn conversations with extensive context, or multi-modal inputs including images, audio, or video. These large payloads require different buffering, streaming, and timeout strategies than traditional HTTP APIs.

Resource Consumption Patterns

Perhaps most critically, a single inference request can consume an entire GPU’s resources during processing. This is fundamentally different from traditional request serving where multiple requests can be processed concurrently on the same compute resources. When a GPU is fully engaged with one request, additional requests must queue, making the scheduling and routing decision far more impactful than those for standard API workloads.

This resource exclusivity means that simple round-robin or least-connection algorithms can create severe imbalances. Sending requests to a server that’s already processing a complex inference task doesn’t just add latency, it can cause resource contention that impacts performance for all queued requests.

Stateful Considerations and Memory Management

AI models often maintain in-memory caches that significantly impact performance. KV caches store intermediate attention calculations for previously processed tokens, serving as the primary consumer of GPU memory during generation and often becoming the most common bottleneck. When KV cache utilization approaches limits, performance degrades dramatically, making cache-aware routing essential.

Additionally, many modern AI deployments use fine-tuned adapters like LoRA (Low-Rank Adaptation) to customize model behavior for specific users, organizations, or use cases. These adapters consume GPU memory and loading time when switched. A model server that already has the required LoRA adapter loaded can process requests immediately, while servers without the adapter face expensive loading overhead that can take seconds to complete.

Queue Dynamics and Criticality

AI inference workloads also introduce the concept of request criticality that’s less common in traditional services. Real-time interactive applications (like chatbots or live content generation) require low latency and should be prioritized, while batch processing jobs or experimental workloads can tolerate higher latency or even be dropped during system overload.

Traditional load balancers lack the context to make these criticality-based decisions. They can’t distinguish between a time-sensitive customer support query and a background batch job, leading to suboptimal resource allocation during peak demand periods.

This is where inference-aware routing becomes critical. Instead of treating all backends as equivalent black boxes, we need routing decisions that understand the current state and capabilities of each model server, including their queue depth, memory utilization, loaded adapters, and ability to handle requests of different criticality levels.

Gateway API Inference Extension: A Kubernetes-Native Solution

The Kubernetes Gateway API Inference Extension has introduced solutions to these challenges, building on the proven foundation of Kubernetes Gateway API while adding AI-specific intelligence. Rather than requiring organizations to patch together custom solutions or abandon their existing Kubernetes infrastructure, the extension provides a standardized, vendor-neutral approach to intelligent AI traffic management.

The extension introduces two key Custom Resource Definitions that work together to address the routing challenges we’ve outlined. The InferenceModel resource provides an abstraction for AI-Inference workload owners to define logical model endpoints, while the InferencePool resource gives platform operators the tools to manage backend infrastructure with AI workload awareness.

By extending the familiar Gateway API model rather than creating an entirely new paradigm, the inference extension enables organizations to leverage their existing Kubernetes expertise while gaining the specialized capabilities that AI workloads demand. This approach ensures that teams can adopt intelligent inference routing aligned with familiar networking knowledge and tooling.

Note: InferenceModel is likely to change in future Gateway API Inference Extension releases.

InferenceModel

The InferenceModel resource allows inference workload owners to define logical model endpoints that abstract the complexities of backend deployment.

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: customer-support-bot
  namespace: ai-workloads
spec:
  modelName: customer-support
  criticality: Critical
  poolRef:
    name: llama-pool
  targetModels:
    - name: llama-3-8b-customer-v1
      weight: 80
    - name: llama-3-8b-customer-v2
      weight: 20

This configuration exposes a customer-support model that intelligently routes between two backend variants, enabling safe rollouts of new model versions while maintaining service availability.

InferencePool

The InferencePool acts as a specialized backend service that understands AI workload characteristics:

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llama-pool
  namespace: ai-workloads
spec:
  targetPortNumber: 8000
  selector:
    app: llama-server
    version: v1
  extensionRef:
    name: llama-endpoint-picker

When integrated with Istio, this pool automatically discovers model servers through Istio’s service discovery.

How Inference Routing Works in Istio

Istio’s implementation builds on the service mesh’s proven traffic management foundation. When a request enters the mesh through a Kubernetes Gateway, it follows the standard Gateway API HTTPRoute matching rules. However, instead of using traditional load balancing algorithms, the backend is picked by an Endpoint Picker (EPP) service.

The EPP evaluates multiple factors to select the optimal backend:

Request Criticality Assessment: Critical requests receive priority routing to available servers, while lower criticality requests (Standard or Sheddable) may be load-shed during high utilization periods.
Resource Utilization Analysis: The extension monitors GPU memory usage, particularly KV cache utilization, to avoid overwhelming servers that are approaching capacity limits.
Adapter Affinity: For models using LoRA adapters, requests are preferentially routed to servers that already have the required adapter loaded, eliminating expensive loading overhead.
Prefix-Cache Aware Load Balancing: Routing decisions consider distributed KV cache states across model servers, and prioritize model servers that already have the prefix in their cache.
Queue Depth Optimization: By tracking request queue lengths across backends, the system avoids creating hotspots that would increase overall latency.

This intelligent routing operates transparently within Istio’s existing architecture, maintaining compatibility with features like mutual TLS, access policies, and distributed tracing.

Inference Routing Request Flow

Flow of an inference request with gateway-api-inference-extension routing.

The Road Ahead

The future roadmap includes istio-related features such as:

Support for Waypoints - As Istio continues to evolve toward ambient mesh architecture, inference-aware routing will be integrated into waypoint proxies to provide centralized, scalable policy enforcement for AI workloads.

Beyond Istio-specific innovations, the Gateway API Inference Extension community is also actively developing several advanced capabilities that will further enhance routing for for AI inference workloads on Kubernetes:

HPA Integration for AI Metrics: Horizontal Pod Autoscaling based on model-specific metrics rather than just CPU and memory.
Multi-Modal Input Support: Optimized routing for large multi-modal inputs and outputs (images, audio, video) with intelligent buffering and streaming capabilities.
Heterogeneous Accelerator Support: Intelligent routing across different accelerator types (GPUs, TPUs, specialized AI chips) with latency and cost-aware load balancing.

Getting Started with Istio Inference Extension

Ready to try inference-aware routing? The implementation is officially available starting with Istio 1.27!

For installation and guides, please follow the Istio-specific guidance on the Gateway API Inference Extension website.

Performance Impact and Benefits

Early evaluations show significant performance improvements with inference-aware routing, including substantially lower p90 latency at higher query rates and reduced end-to-end tail latencies compared to traditional load balancing.

For detailed benchmark results and methodology, see the Gateway API Inference Extension performance evaluation with testing data using H100 GPUs and vLLM deployments.

The integration with Istio’s existing infrastructure means these benefits come with minimal operational overhead, and your existing monitoring, security, and traffic management configurations continue to work unchanged.

Conclusion

The Gateway API Inference Extension represents a significant step forward in making Kubernetes truly AI-ready, and Istio’s implementation brings this intelligence to the service mesh layer where it can have maximum impact. By combining inference-aware routing with Istio’s proven security, observability, and traffic management capabilities, we’re enabling organizations to run AI workloads with the same operational excellence they expect from their traditional services.

Have a question or want to get involved? Join the Kubernetes Slack and then find us on the #gateway-api-inference-extension channel or discuss on the Istio Slack.