How MaaS serves internal vLLM models and external providers behind one Envoy gateway with API-key authentication, token-based rate limiting, and body-based model routing on RHOAI 3.5
Every inference request passes the same Envoy filter chain. On RHOAI 3.5 two external processors bracket the Kuadrant policy shim: ipp-pre reads the model name out of the body before auth, so OpenAI-standard POST /v1/chat/completions works with no path prefix; ipp runs after policy and rewrites the request for the backend. Traffic never leaves Envoy for a detour — Authorino and Limitador are gRPC callouts.
What happens, hop by hop, when a client sends POST /v1/chat/completions with {"model":"gpt-5.6-luna"} — the body-based form that any OpenAI SDK produces unmodified. The legacy path form POST /swongpai-vllm/gpt-5.6-luna/v1/chat/completions takes the same route; only the identity lookup in step 4 differs. Filter order below was read from the gateway's Envoy config_dump, not from CRs.
Inference needs a key. Keys are minted by maas-api — which on 3.5 lives in its own namespace, redhat-ai-gateway-infra — scoped to a subscription, and stored in PostgreSQL. The subscription, not the key, decides which models you reach and how many tokens you get. Max lifetime 90 days; expired keys are swept every 15 minutes.
Resources grouped by namespace, as they exist on 3.5. Two namespaces are new since 3.4: redhat-ai-gateway-infra (maas-api moved here) and ai-tenants (the AITenant that binds a tenant namespace to a Gateway).
Intent layer. maas-controller compiles the two policies into one gateway AuthPolicy allow-map and one TokenRateLimitPolicy per model route.
Every component in the request path and control plane, by layer — with live images, ports, and role. Pulled from the running 3.5 cluster on 2026-09-04.
| Component | Ports | Role | Image |
|---|---|---|---|
| AWS NLB + Route53 | 443 | Cloud entry point. swongpai-maas.apps.… → NLB → gateway pod. Created by OpenShift when the Gateway was applied. | — |
| maas-default-gateway (istio-proxy) | 443 TLS15021 health | The Envoy data plane. Terminates TLS, runs ipp-pre → Kuadrant wasm → ipp → router, matches HTTPRoutes. Everything flows through this pod. | openshift-service-mesh/istio-proxyv2-rhel9 |
| istiod-openshift-gateway | 150101501215014 | OpenShift 4.20's built-in Gateway API control plane. Translates Gateway/HTTPRoute/EnvoyFilter into xDS. No OpenShift Service Mesh operator is installed. | openshift-service-mesh/istio-pilot-rhel9 |
| Component | Ports | Role | Image |
|---|---|---|---|
| payload-pre-processing (ext_proc ipp-pre) | 9004 grpc | New in 3.5. Runs before policy. Extracts .model from the request body into X-Gateway-Model-Name, resolves the provider header, clears the route cache. Fail-open. Enables body-based routing. | rhoai/odh-ai-gateway-payload-processing-rhel9 |
| authorino | 50051 grpc5001 http | External authorization for the gateway-scoped maas-gateway-auth. Validates sk-oai-* keys via maas-api, does k8s TokenReview for everything else, runs three OPA/Rego checks, caches 60s. | rhcl-1/authorino-rhel9 (1.4.2) |
| limitador-limitador | 8081 grpc8080 http | Rate-limit service. One token counter per userid per subscription window, compiled from every TokenRateLimitPolicy. Debited post-response by real usage. | rhcl-1/limitador-rhel9 (1.4.1) |
| payload-processing (ext_proc ipp) | 9004 grpc | Runs after policy. Request side: header guard, stream-usage enforcer, provider resolution, API translation, provider-key injection. Response side: extracts usage.total_tokens for Limitador. Fail-closed — outage stops all gateway traffic. | rhoai/odh-ai-gateway-payload-processing-rhel9 |
| Component | Ports | Role | Image |
|---|---|---|---|
| maas-controller | — | Operator. Bootstraps Config → AITenant → MaasTenantConfig, deploys maas-api into --infra-namespace, compiles MaaSAuthPolicy/MaaSSubscription/MaaSModelRef into the gateway AuthPolicy allow-map and per-route TRLPs, owns the payload-processing EnvoyFilter and the Limitador ServiceMonitor. | rhoai/odh-maas-controller-rhel9 |
| ai-gateway-operator | — | New in 3.5. Platform operator for the AI Gateway layer (payload processing images, gateway integration). Runs in redhat-ods-applications alongside maas-controller. | rhoai/odh-ai-gateway-operator-rhel9 |
| maas-api (redhat-ai-gateway-infra) | 8443 https9090 metrics | REST API for key lifecycle and the model catalog (/v1/models). Checks subscription owner groups before minting. Authorino's key-validation backend. Ships SSL_CERT_DIR for the service-ca — the 3.4 env patch is gone. Reached through HTTPRoute/maas-api-route on the MaaS gateway. | rhoai/odh-maas-api-rhel9 |
| postgres-ha-app (CNPG) | 5432 | 3-instance PostgreSQL 18.4 in maas-postgres; 2 healthy today (instance 2's PV is in AZ 1c, both nodes are in 1a). Reached via postgres-ha-app-rw, sslmode=require. | cloudnative-pg/postgresql:18.4 · CNPG 1.30.0 |
| maas-ui (dashboard module) | 8243 | MaaS panel in the RHOAI dashboard — self-service keys and catalog. Separate Deployment owned by Dashboard/default-dashboard on 3.5. | rhoai/odh-mod-arch-maas-rhel9 |
| Component | Ports | Role | Image |
|---|---|---|---|
| llmisvc-controller-manager | — | KServe's LLMInferenceService controller. Expands one CR into vLLM Deployment, Services, HTTPRoute (and EPP + InferencePool when router.scheduler is set — not on the current three). 3.5 adds v3-5-0-* presets incl. single/multi-node and prefill/decode templates. | rhoai/odh-kserve-llmisvc-controller-rhel9 |
| vLLM pods (×3 defined, 0 running) | 8000 http | muse-glimmer-llmisvc-nightly (pvc, Muse-Glimmer-30B-NVFP4), qwen38-27b-018 (oci modelcar-catalog), redhataiqwen3-coder-next-nvfp4 (oci registry.redhat.io/rhai). 1 GPU each; two are annotated serving.kserve.io/stop. Need the H100 node. | rhaii/vllm-cuda-rhel9 + modelcar |
| ExternalProvider openai-prod + Service / ServiceEntry / DestinationRule | → 443 | New in 3.5. inference.opendatahub.io CRD. One per upstream: ExternalName Service to api.openai.com, ServiceEntry (MESH_EXTERNAL, DNS), DestinationRule (TLS SIMPLE). Shared by every ExternalModel on that provider. | — (CRD) |
| ExternalModel gpt-5.6-luna | — | Object name = modelName = targetModel = upstream id — the one naming rule. Owns HTTPRoute/gpt-5.6-luna (4 rules: path-based and header-based, each with and without the provider header). Body is forwarded to OpenAI untouched. | — (CRD) |
| OGXServer lsd-genai-playground | 8321 | GenAI Studio backend. Replaces the 3.4 LlamaStackDistribution (DSC llamastackoperator: Removed, ogx: Managed). VLLM_TLS_VERIFY=false for intra-namespace vLLM; pgvector for RAG. | rhoai/odh-ogx-core-rhel9 |
| MCPServer rh-demo-mcp-ocp | 8080 /mcp | OpenShift MCP server (Tech Preview) managed by mcp-gateway 0.7.1 + DSC mcplifecycleoperator. Not registered on the MaaS gateway yet. | openshift-mcp-tech-preview/openshift-mcp-server-rhel9:0.4 |
| Component | Ports | Role | Image |
|---|---|---|---|
| Prometheus (MonitoringStack) | 9090 | Scrapes gateway / Limitador (Config-owned ServiceMonitor, 30s) / vLLM metrics with the TelemetryPolicy's labels. 5Gi PV, 90-day retention, Thanos sidecar. | cluster-observability-operator/prometheus-rhel9 |
| OpenTelemetry Collector | — | Statefulset 2/2 + target allocator (0.152.1). Metrics and traces from RHOAI components → Prometheus / Tempo. | rhosdt/opentelemetry-collector-rhel9 |
| Tempo (monolithic) | — | Tracing backend, Tempo 2.10.5. PV storage, 2160h retention, 0.1 sample ratio from the DSCI. | rhosdt/tempo-rhel9 |
| Perses | — | Renders 8 dashboards in the RHOAI console, including dashboard-3-maas-usage-admin (shipped by maas-controller) and the llm-d traffic / utilization / performance set. Tech Preview. | cluster-observability-operator/perses-rhel9 |
| Alertmanager + Thanos Querier | — | Alert routing (2 replicas) and unified cross-shard queries for the dashboard proxies. | cluster-observability-operator/alertmanager-rhel9, thanos-rhel9 |
llm-d is a Kubernetes-native distributed inference framework (Red Hat, Google, CoreWeave, IBM, NVIDIA — built on vLLM and the Gateway API Inference Extension). RHOAI 3.5 ships it as the engine behind every LLMInferenceService. Its core idea: the load balancer should understand LLMs.
Provenance. This section was recorded on 2026-07-06 against redhataillama-31-8b-instruct (2 replicas, namespace swongpai-maas) when the H100 was up and the LLMInferenceService set router.scheduler. The three models defined today run router.gateway + router.route only — no EPP, no InferencePool, plain Service routing — and the GPU node is scaled down. The mechanics are unchanged on 3.5 and 3.5 adds v3-5-0-* presets for single/multi-node and prefill/decode; add spec.router.scheduler: {} to an LLMInferenceService to get the picture below.
Round-robin and least-connections assume all requests cost roughly the same and any backend can serve them equally well. Both assumptions break for LLMs:
llm-d's answer: pull the routing decision out of the proxy and give it to a model-aware sidecar service — the Endpoint Picker (EPP). Envoy still moves the bytes; the EPP decides where.
The EPP runs as the …-kserve-router-scheduler pod (3 containers: scheduler, tokenizer, modelcar). Envoy holds each request and consults it over gRPC :9002. Here is the decision pipeline as it was configured in July:
Worked example — a user sends the 3rd turn of a chat session; the 2,400-token conversation prefix is cached on Pod 1, which is currently busier (illustrative scores, 0–1 normalized):
| Pod | queue-scorer (×2) | prefix-cache-scorer (×3) | Total |
|---|---|---|---|
| Pod 1 — busy, cache warm | 0.20 × 2 = 0.40 | 0.95 × 3 = 2.85 | 3.25 |
| Pod 2 — idle, cache cold | 0.90 × 2 = 1.80 | 0.00 × 3 = 0.00 | 1.80 |
The EPP sends the request to the busier pod — deliberately. Reusing 2,400 cached tokens skips the entire prefill for that prefix, cutting time-to-first-token far more than a shorter queue would. That trade-off is exactly what the 3-vs-2 weighting encodes. If no pod has a cache hit, prefix scores are all zero and queue depth decides — graceful fallback to load-based balancing.
Full request lifecycle through the EPP:
If the EPP itself dies: the InferencePool is configured failureMode: FailOpen — Envoy falls back to cache-blind balancing across the pool. Latency degrades; traffic survives.
apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: EndpointPickerConfig
plugins:
- type: single-profile-handler
- type: queue-scorer
- type: prefix-cache-scorer
- type: max-score-picker
schedulingProfiles:
- name: default
plugins:
- pluginRef: queue-scorer
weight: 2
- pluginRef: prefix-cache-scorer
weight: 3
- pluginRef: max-score-picker
apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
name: redhataillama-31-8b-instruct-inference-pool
namespace: swongpai-maas
spec:
endpointPickerRef:
failureMode: FailOpen # if EPP dies, fall back to normal routing
kind: Service
name: redhataillama-31-8b-instruct-epp-service
port:
number: 9002
selector:
matchLabels:
app.kubernetes.io/name: redhataillama-31-8b-instruct
kserve.io/component: workload
targetPorts:
- number: 8000
llm-d defines three "well-lit paths" — proven deployment profiles in increasing order of scale. This cluster runs the first. Evidence the higher rungs are already wired in: the LLMInferenceService status carries preset annotations for config-llm-prefill-template and config-llm-decode-worker-data-parallel — unused at replicas: 2, but one spec change away.
Identical replicas + EPP with queue & prefix-cache scorers. Lower TTFT and higher goodput than any cache-blind balancer, with no change to the model pods themselves. Fits single-node and modest multi-GPU setups.
Split the two phases onto different pods: prefill is compute-bound (long prompts), decode is memory-bandwidth-bound (token streaming). Specialized pods each run at their bottleneck; computed KV tensors are transferred prefill→decode over a fast interconnect. Pays off for long-context, high-QPS serving.
For giant Mixture-of-Experts models (DeepSeek-R1 class): experts sharded across many GPUs/nodes with data-parallel attention, deployed as multi-node leader/worker groups. The EPP's role grows to routing across heterogeneous multi-node serving groups.
Every pod runs both prefill and decode. The EPP scores replicas by queue depth and prefix-cache state, routing each request to the optimal pod. This is the default when your LLMInferenceService has no spec.prefill and no spec.worker — just spec.replicas and spec.template.
Core trade-off: Simplicity and resilience vs. potential prefill interference. When a pod is busy with a long prefill (processing a 10k-token prompt), its in-flight decode requests stall — every user sharing that pod feels the latency spike. For small models and moderate QPS, this rarely matters. For long-context, high-throughput workloads, it becomes the bottleneck.
| Pros | Cons |
|---|---|
| Simplest config — just set replicas | Prefill interference: long prompts stall decode on the same pod |
| Every pod is identical — any can serve any request | Cannot optimize hardware separately for compute-heavy prefill vs. bandwidth-heavy decode |
| EPP prefix-cache routing already reduces redundant prefill | At high QPS, tail latency grows as prefill contention increases |
| FailOpen — survives EPP death gracefully | KV cache limited to local GPU memory per pod |
| Works on single-node / small GPU counts |
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
name: redhataillama-31-8b-instruct
namespace: swongpai-maas
spec:
model:
uri: oci://registry.redhat.io/rhelai1/modelcar-llama-3-1-8b-instruct:1.5
replicas: 2 # both do prefill + decode
# no spec.prefill → unified mode
# no spec.worker → single-node
template:
containers:
- name: main
resources:
limits: {nvidia.com/gpu: "1", cpu: "2", memory: "4Gi"}
router:
gateway:
refs: [{name: maas-default-gateway, namespace: openshift-ingress}]
route: {}
scheduler: {}
Best for: Models ≤ 70B, moderate QPS (<50 req/s), short-to-medium context (<8k tokens), single GPU node, getting started fast.
Variant — no scheduler at all: omit router.scheduler from the spec and no EPP pod is deployed; requests load-balance through a plain Kubernetes Service (kube-proxy). Lowest overhead, but loses prefix-cache routing, queue awareness, and any P/D capability. Fine for dev/testing; production should keep the scheduler.
| Feature | No Scheduler | Default Scheduler (this cluster) | Prefill/Decode |
|---|---|---|---|
| Routing logic | k8s Service | EPP load balancing | EPP + P/D separation |
| Prefix-cache routing | ✗ | ✓ | ✓ |
| KV cache transfer | ✗ | ✗ | ✓ (NIXL/RDMA) |
| Resource overhead | Lowest | Low | Medium |
| Use case | Dev / simple | Production (basic) | Production (advanced) |
Adding spec.prefill tells the controller to split inference into two specialized pod pools. Prefill pods crunch the full prompt (compute-bound — all tokens processed in parallel), then transfer the computed KV tensors to a decode pod via the NIXL v2 connector (llm-d-routing-sidecar with --kv-connector=nixlv2). The decode pod skips prefill entirely and begins streaming tokens immediately.
Core trade-off: Eliminates prefill-decode interference completely, but adds KV transfer latency and operational complexity. You now have two pools to scale, two sets of resources to right-size, and a dependency on the KV transfer channel. The decode template on your cluster already bundles the llm-d-routing-sidecar init container with NIXL v2 — it just activates when spec.prefill appears.
| Pros | Cons |
|---|---|
| Zero prefill interference — decode latency is stable regardless of prompt length | KV transfer adds latency (RDMA ~100μs, TCP ~1-5ms per transfer) |
| Each pool optimized for its bottleneck: prefill=compute, decode=memory bandwidth | Two deployments to manage, scale, and monitor |
| Independent scaling — scale prefill for prompt-heavy, decode for streaming-heavy | Needs RDMA/RoCE for best performance (TCP fallback exists but loses the speed advantage) |
| Better GPU utilization — prefill pods can use chunked-prefill (--enable-chunked-prefill) | Minimum 2 GPUs (1 prefill + 1 decode) — no replica redundancy without 3+ |
| Lower inter-token latency (ITL) at high QPS | Debug surface grows: sidecar, NIXL channel, two pools of pods |
Not all-or-nothing — the threshold parameter: the scheduler config decides per request whether to use the prefill pool. threshold: 0 = every request goes through P/D separation; threshold: N = only prompts with estimated tokens > N take the prefill-pool detour — short prompts are prefilled and decoded entirely on the decode pod, skipping the KV transfer cost. Routing rule: new request without cached KV → prefill pool; continuation with KV already on a decode pod → straight to that decode pod. Pool sizes are independent — the upstream sample runs 2 prefill + 1 decode replicas (prefill was the bottleneck).
RDMA prerequisites (for real KV-transfer performance): a RoCE NetworkAttachmentDefinition, an rdma/roce_gdr resource on each pod, KSERVE_INFER_ROCE=true (the decode template on this cluster already ships the RoCE auto-discovery script that sets NCCL_IB_HCA / UCX_NET_DEVICES), and VLLM_NIXL_SIDE_CHANNEL_HOST pointing at the pod IP. Note: if prefill and decode pods land on the same node, KV transfer rides NVLink/local paths instead of RDMA — pod anti-affinity forces cross-node RDMA when that's what you're benchmarking.
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
name: redhataillama-31-8b-instruct
namespace: swongpai-maas
spec:
model:
uri: oci://registry.redhat.io/rhelai1/modelcar-llama-3-1-8b-instruct:1.5
# ── Decode pods (token generation — latency-sensitive)
replicas: 1
template:
containers:
- name: main
args: ["--enforce-eager"]
resources:
limits: {nvidia.com/gpu: "1", cpu: "2", memory: "4Gi"}
# ── Prefill pods (prompt processing — compute-heavy)
# Presence of spec.prefill triggers disaggregated mode
prefill:
replicas: 1
template:
containers:
- name: main
args: ["--enable-chunked-prefill"]
resources:
limits: {nvidia.com/gpu: "1", cpu: "2", memory: "4Gi"}
router:
gateway:
refs: [{name: maas-default-gateway, namespace: openshift-ingress}]
route: {}
scheduler: {}
# Controller auto-injects from v3-4-2-kserve-config-llm-decode-template:
# llm-d-routing-sidecar initContainer with --kv-connector=nixlv2
# Separate InferencePool per pool type
# From LLMInferenceServiceConfig v3-4-2-kserve-config-llm-decode-template # Controller injects this into decode pods when spec.prefill is present: initContainers: - name: llm-d-routing-sidecar image: registry.redhat.io/rhoai/odh-llm-d-routing-sidecar-rhel9@sha256:18dbf82a… command: [/app/pd-sidecar] args: - --port=8000 # proxy port (replaces vLLM's external port) - --vllm-port=8001 # vLLM actual port (moved behind sidecar) - --kv-connector=nixlv2 # NVIDIA NIXL v2 for KV cache transfer - --enable-ssrf-protection=true - --pool-group=inference.networking.x-k8s.io # TLS args injected when GlobalConfig.EnableTLS is true
Best for: Models 13B+, high QPS (>50 req/s), long context (8k–128k tokens), latency-sensitive decode (chatbots, streaming), clusters with 4+ GPUs and RDMA networking.
For massive Mixture-of-Experts (MoE) models like DeepSeek-R1 (671B, 256 experts). The model is sharded across multiple GPUs/nodes using spec.parallelism (tensor, pipeline, data, expert) and spec.worker (which creates a LeaderWorkerSet — leader + N worker pods that form one serving group). Can be combined with spec.prefill for disaggregated multi-node serving.
Core trade-off: The only way to serve models that don't fit on a single GPU, but it requires multi-node coordination, high-bandwidth interconnects (NVLink/NVSwitch/InfiniBand), and significant operational expertise. The EPP's role grows from "pick a pod" to "pick a serving group" across heterogeneous multi-node clusters.
| Pros | Cons |
|---|---|
| Only way to serve models that exceed single-GPU memory | Requires NVLink/InfiniBand — bandwidth between GPUs becomes the bottleneck |
| Expert parallelism with data-parallel attention — efficient for MoE architectures | Pod scheduling is complex: leader/worker pods must be co-located with RDMA |
| Can combine with P/D disaggregation for maximum throughput | Minimum hardware: 8+ GPUs across multiple nodes |
| Horizontal scaling of truly massive models (200B+) | Significant operational complexity — LeaderWorkerSet, network topology, failure domains |
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
name: deepseek-r1
spec:
model:
uri: oci://…/deepseek-r1-671b:<pinned-tag>
replicas: 1
parallelism:
tensor: 8 # shard across 8 GPUs
data: 1 # data-parallel replicas
expert: true # enable expert parallelism for MoE
worker: # LeaderWorkerSet: 1 leader + N workers
# full PodSpec — co-located worker pods
containers:
- name: main
resources:
limits: {nvidia.com/gpu: "8"}
# Optionally combine with P/D disaggregation:
# prefill:
# replicas: 1
# parallelism:
# tensor: 8
# template: …
Best for: MoE models (DeepSeek, Mixtral 8×22B+), dense 70B+ models that need tensor parallelism, multi-node GPU clusters with high-speed interconnects.
Yes — across models. No — within one model.
Each LLMInferenceService is an independent deployment with its own mode. On the same cluster you can run:
Each gets its own HTTPRoute and MaaS policies. Models with router.scheduler also get their own InferencePool and EPP; models without it use plain Service routing. They share the Gateway and MaaS control plane (maas-api, Authorino, Limitador). Subscriptions and rate limits are per-model — fully independent quota management.
What you cannot do: hybrid replica roles within a single model — e.g., "some replicas do both prefill+decode, some specialize." The CRD is binary at the pod level: spec.prefill present → separate prefill and decode pools. Absent → all unified.
But per-request hybrid exists: in a disaggregated deployment, the scheduler's threshold parameter routes only long prompts (estimated tokens > N) through the prefill pool — short prompts are handled entirely by the decode pod, skipping the KV-transfer round trip. So a P/D deployment with threshold > 0 behaves like unified mode for small requests and disaggregated mode for large ones — the closest thing to "mixed" within one model.
The machinepool is at zero today. After respin and MIG relabeling, the node exposes exactly 2 GPU slots:
| Configuration | Slot 1 | Slot 2 | Mode | Viable? |
|---|---|---|---|---|
| Historical July: Llama 8B × 2 unified | vLLM replica 1 | vLLM replica 2 | Unified | ✓ Fit verified in July; not running today |
| Llama 8B disaggregated | Prefill pod | Decode pod | P/D | ⚠ Works, but no redundancy and no RDMA benefit (same node) |
| Two different models unified | Model A × 1 | Model B × 1 | Mixed unified | ✓ Both ≤ 40GB VRAM each |
| One model unified + one disaggregated | Not enough slots — disaggregation needs ≥ 2 GPUs for one model | — | ✗ Need 3+ GPUs | |
| Any multi-node / expert mode | Requires multiple nodes with high-speed interconnect | — | ✗ Single node | |
Practical recommendation for this cluster: With only two slices after respin, use unified serving unless a measured workload justifies spending both slices on one disaggregated model. Two one-GPU models can coexist as independent LLMInferenceServices behind the same Gateway. Add router.scheduler only when EPP scheduling is needed.
The mechanics behind the diagrams — with the actual 3.5 configuration from the live cluster, collapsible below each explanation.
The platform chain (operator → infra): Setting kserve.modelsAsService: Managed in the DSC makes maas-controller bootstrap a Config/default, which owns an AITenant (tenant namespace ↔ Gateway), a MaasTenantConfig, the maas-api Deployment in redhat-ai-gateway-infra, its HTTPRoute, the payload-processing EnvoyFilter and a Limitador ServiceMonitor. The 3.4 Tenant is still there, annotated deprecated, and ignored.
The intent chain (humans → policy): Admins write MaaSAuthPolicy (who may reach which models), MaaSSubscription (how many tokens), MaaSModelRef (which model). maas-controller compiles all auth policies into one gateway-scoped Kuadrant AuthPolicy with an OPA allow-map, and each subscription into a TokenRateLimitPolicy on the model's HTTPRoute.
The serving chain (model → backend): Internal — an LLMInferenceService is expanded by llmisvc-controller into vLLM Deployment, Services and HTTPRoute. External — an ExternalProvider + ExternalModel pair yields an ExternalName Service, ServiceEntry, DestinationRule and HTTPRoute. Either way the chains meet only at the HTTPRoute.
Three gateway-wide policies enforce a zero-trust posture; on 3.5 only the first is what actually decides:
Adding a model therefore adds exactly two keys to one allow-map (<name> and <ns>/<name>) and one TRLP — everything else stays shut. Deleting a model does not prune its allow-map key.
An external model's route matches on headers that ipp-pre injected, so the client never needs a path prefix. An internal model's route is the classic prefix-and-rewrite. Both attach to the same Gateway and both are policy targets.
parentRefs:
- {kind: Gateway, name: maas-default-gateway, namespace: openshift-ingress}
rules:
# body-based: header set by ipp-pre, provider resolved
- matches:
- path: {type: PathPrefix, value: /}
headers:
- {name: X-Gateway-Model-Name, type: Exact, value: gpt-5.6-luna}
- {name: x-ipp-selected-provider, type: Exact, value: openai-prod}
filters:
- type: RequestHeaderModifier
requestHeaderModifier: {set: [{name: Host, value: api.openai.com}]}
backendRefs:
- {kind: Service, name: openai-prod, port: 443} # ExternalName → api.openai.com
timeouts: {request: 300s}
# path-based (legacy): /swongpai-vllm/gpt-5.6-luna, with and without the provider header
- matches:
- path: {type: PathPrefix, value: /swongpai-vllm/gpt-5.6-luna}
headers: [{name: x-ipp-selected-provider, type: Exact, value: openai-prod}]
# … same filters / backend / timeout …
# + the two header-less fallbacks of each shape (4 rules total). The path is rewritten
# to /v1/chat/completions by ipp's api-translation stage, not by the route.
rules:
# four OpenAI-style APIs × two path forms, each rewritten to the bare API path
- matches: [{path: {type: PathPrefix, value: /swongpai-vllm/muse-glimmer-llmisvc-nightly/v1/chat/completions}}]
filters: [{type: URLRewrite, urlRewrite: {path: {type: ReplacePrefixMatch, replacePrefixMatch: /v1/chat/completions}}}]
backendRefs: [{kind: Service, name: muse-glimmer-llmisvc-nightly-kserve-workload-svc, port: 8000}]
timeouts: {request: 0s, backendRequest: 0s} # streaming: no timeout
# identical rules for /v1/completions, /v1/responses, /v1/messages …
# and for the 3.5 catalog form:
- matches: [{path: {type: PathPrefix, value: /publishers/swongpai-vllm/models/muse-glimmer-llmisvc-nightly/v1/chat/completions}}]
# … same rewrite / backend …
# catch-all for both prefixes → / on the workload Service
- matches: [{path: {type: PathPrefix, value: /swongpai-vllm/muse-glimmer-llmisvc-nightly}}]
filters: [{type: URLRewrite, urlRewrite: {path: {type: ReplacePrefixMatch, replacePrefixMatch: /}}}]
backendRefs: [{kind: Service, name: muse-glimmer-llmisvc-nightly-kserve-workload-svc, port: 8000}]
No InferencePool appears here: the three current LLMInferenceServices set router.gateway + router.route only. Section 06 describes what changes when router.scheduler is added.
One AuthPolicy on the Gateway. Two credential types: API keys matching ^Bearer sk-oai-.* (priority 0) and any other bearer token via Kubernetes TokenReview (priority 2). Three authorization rules, the middle one carrying the compiled allow-map. Cache keys include the model identity, so a verdict is per user × model for 60s.
targetRef: {group: gateway.networking.k8s.io, kind: Gateway, name: maas-default-gateway}
defaults:
rules:
authentication:
api-keys:
plain: {selector: request.headers.authorization}
priority: 0
when: [{selector: request.headers.authorization, operator: matches, value: ^Bearer sk-oai-.*}]
openshift-identities:
kubernetesTokenReview: {audiences: [https://oidc.op1.openshiftapps.com/…]}
priority: 2
when: [{predicate: '!request.headers.authorization.startsWith("Bearer sk-oai-")'}]
authorization:
auth-valid: # key was validated by maas-api, or it is not a key at all
cache: {ttl: 60}
opa:
rego: |
allow { input.auth.metadata.apiKeyValidation.valid == true }
allow { not input.auth.metadata.apiKeyValidation }
deny-client-identity-headers: # clients may not forge MaaS identity
patternMatching:
patterns:
- {predicate: '!("x-maas-username" in request.headers)'}
- {predicate: '!("x-maas-group" in request.headers)'}
require-group-membership:
cache: {ttl: 60}
opa:
rego: |
model_access := {
"gpt-5.6-luna": {"groups": ["cluster-admins"]},
"swongpai-vllm/gpt-5.6-luna": {"groups": ["cluster-admins"]},
"swongpai-vllm/muse-glimmer-llmisvc-nightly": {"groups": ["cluster-admins"]},
"publishers/swongpai-vllm/models/muse-glimmer-llmisvc-nightly": {"groups": ["cluster-admins"]},
"swongpai-vllm/redhataiqwen3-coder-next-nvfp4": {"groups": ["cluster-admins"]},
}
path_parts := [p | p := split(request_path, "/")[_]; p != ""]
path_model_identity := sprintf("%s/%s", [path_parts[0], path_parts[1]]) {
count(path_parts) >= 2; path_parts[0] != "v1"; path_parts[0] != "maas-api"
}
header_model_identity := object.get(request_headers, "x-gateway-model-name", "")
model_identity := path_model_identity else header_model_identity
# allow when the caller's groups intersect model_access[model_identity].groups
targetRef: {group: gateway.networking.k8s.io, kind: HTTPRoute, name: gpt-5.6-luna}
limits:
models-as-a-service-gpt-5-6-luna-subscription-gpt-5.6-luna-tokens:
counters: [{expression: auth.identity.userid}]
rates: [{limit: 100000, window: 1h}]
when:
- predicate: auth.identity.selected_subscription_key ==
"models-as-a-service/gpt-5-6-luna-subscription@swongpai-vllm/gpt-5.6-luna"
&& !request.path.endsWith("/v1/models")
models-as-a-service-sa-premium-gpt-5.6-luna-tokens:
counters: [{expression: auth.identity.userid}]
rates: [{limit: 90000, window: 1s}] # mint external keys against the other subscription
when:
- predicate: auth.identity.selected_subscription_key ==
"models-as-a-service/sa-premium@swongpai-vllm/gpt-5.6-luna"
&& !request.path.endsWith("/v1/models")
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: maas-default-gateway
namespace: openshift-ingress
annotations:
security.opendatahub.io/authorino-tls-bootstrap: "true"
argocd.argoproj.io/tracking-id: openshift-ai-maas-config:… # owned by Surote/ocp-configuration
spec:
gatewayClassName: openshift-default
listeners:
- name: https
port: 443
protocol: HTTPS
hostname: swongpai-maas.apps.rosa.rsaijp.na2y.p3.openshiftapps.com
tls:
mode: Terminate
certificateRefs: [{kind: Secret, name: maas-gateway-tls}]
allowedRoutes:
namespaces: {from: All}
# status: Programmed=True · attachedRoutes: 3
# address: a2d5644ffe0094200ac4bb0f2ec8f0b2-…elb.amazonaws.com (AWS NLB)
# owned by Config/default · priority: 10
configPatches:
- applyTo: HTTP_FILTER
match: {listener: {filterChain: {filter: {name: envoy.filters.network.http_connection_manager,
subFilter: {name: extensions.istio.io/wasmplugin/openshift-ingress.kuadrant-maas-default-gateway}}}}}
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ext_proc.ipp-pre
typed_config:
failure_mode_allow: true
grpc_service: {envoy_grpc: {cluster_name: outbound|9004||payload-pre-processing.openshift-ingress.svc.cluster.local}}
processing_mode: {request_header_mode: SEND, request_body_mode: FULL_DUPLEX_STREAMED,
response_header_mode: SKIP, response_body_mode: NONE}
- applyTo: HTTP_FILTER
match: {… same anchor …}
patch:
operation: INSERT_AFTER
value:
name: envoy.filters.http.ext_proc.ipp
typed_config:
failure_mode_allow: false
grpc_service: {envoy_grpc: {cluster_name: outbound|9004||payload-processing.openshift-ingress.svc.cluster.local}}
processing_mode: {request_body_mode: FULL_DUPLEX_STREAMED, response_body_mode: FULL_DUPLEX_STREAMED, …}
# + duplicate patches anchored on envoy.filters.http.wasm (covers both anchor spellings — the 3.4 skew fix)
# + HTTP_ROUTE MERGE patches disabling both filters on the maas-api routes
#
# Verify it took (Istio drops a non-matching patch silently):
# oc exec -n openshift-ingress <gateway-pod> -- pilot-agent request GET config_dump
# → ext_proc.ipp-pre, wasm, ext_proc.ipp, router
The H100 machinepool is at zero right now to save cost; only two c5.4xlarge workers remain. What persists is the plan: the GPU Operator (26.3.3, mig.strategy: single) and NFD stay installed, and on respin the node gets nvidia.com/mig.config=all-3g.40gb — two isolated 3g.40gb slices that surface as two plain nvidia.com/gpu. Each LLMInferenceService asks for one. The label is per node and must be re-applied after any machinepool replacement.
Three 1-GPU models are defined for two slices — on respin, two run and one stays serving.kserve.io/stop: "true". Until then every LLMInferenceService is Stopped / MinimumReplicasUnavailable, their MaaSModelRefs Pending, and sa-premium reports Degraded. None of that is a fault.
| Component down | Effect on traffic |
|---|---|
| Envoy gateway pod | Total outage for the MaaS hostname until the Deployment reschedules. NLB health check (:15021) stops routing immediately. |
| payload-pre-processing (ipp-pre) | failure_mode_allow: true — requests continue, but no X-Gateway-Model-Name is set. Body-based calls 404 (no route match); path-based calls keep working. |
| Authorino | All requests fail auth → 5xx from the wasm shim. Nothing reaches models. The 60s verdict cache does not help new connections. |
| Limitador | Rate-limit checks fail; requests may pass unlimited or be denied depending on shim failure mode. Token accounting stops. |
| payload-processing (ipp) | All gateway traffic fails — failure_mode_allow: false and on 3.5 the filter is genuinely in the chain (it was not on 3.4.x). Deliberate: no silent loss of token accounting or provider-key injection. This is the live single point of failure to watch. |
| api.openai.com / provider key | External model only. Bad or missing Secret → 500 credentials not found; upstream outage → 5xx passed through. Internal models unaffected. |
| One vLLM pod | Its route's Service loses the endpoint; with one replica the model is down until rescheduled. (With an EPP, traffic would fail over across replicas.) |
| maas-api | No new keys, no key validation → API-key auth fails as caches expire. Kubernetes-token access keeps working (TokenReview does not touch maas-api). |
| PostgreSQL (all instances) | maas-api loses state — key ops fail. Today 2 of 3 CNPG instances are up (AZ pinning); primary + one replica is still HA. |
| maas-controller / llmisvc-controller | Data plane unaffected. CR changes stop reconciling — existing allow-map, TRLPs and pods keep running as-is. |
| Prometheus / Tempo / Perses | Zero traffic impact. Dashboards and usage attribution go blind until restored. |