OpenShift Cluster Manage Network - VIP and LB Architecture

hj4h5.dynamic2.redhatworkshops.io — OCP 4.20 — Bare Metal IPI

API path

App path

Active

Standby

VIP / keepalived

☁

Client

External Traffic

☲

DNS → NAT

api.hj4h5...redhatworkshops.io

☲

DNS → NAT

*.apps.hj4h5...redhatworkshops.io

↓

API VIP

192.168.37.201

● master2 (active)

Ingress VIP

192.168.37.202

● master2 (active)

nftables DNAT on VIP holder

prerouting: 192.168.37.201:6443 → dnat to 192.168.37.201:9445
output: 192.168.37.201:6443 → redirect to :9445

14,936 external packets + 2,899 local packets redirected — click for full nft rules

↓

Control Plane Nodes

master0

192.168.37.21

master

keepalived

API + Ingress VRRP

KNI HAProxy

:9445 → :6443

Router HAProxy

:80/:443 HostNetwork

kube-apiserver

:6443

CoreDNS

:53

master1

192.168.37.22

master

keepalived

API + Ingress VRRP

KNI HAProxy

:9445 → :6443

Router HAProxy

:80/:443 HostNetwork

kube-apiserver

:6443

CoreDNS

:53

master2

192.168.37.23

master

keepalived

API + Ingress VRRP

KNI HAProxy

:9445 → :6443

Router HAProxy

:80/:443 HostNetwork

kube-apiserver

:6443

CoreDNS

:53

Worker Nodes

worker0

192.168.37.24

worker

keepalived

Ingress VRRP only

CoreDNS

:53

worker1

192.168.37.25

worker

keepalived

Ingress VRRP only

CoreDNS

:53

↓backend pods

pod/app-1

pod/app-2

pod/app-3

⚠ Throughput Insight

Ingress VIP is active/passive via keepalived. Only 1 of 3 router pods receives traffic at any time. Effective app throughput: maxconn 50,000 (single pod), not 150,000. Other 2 router pods are hot standby. API traffic is different — KNI HAProxy on VIP holder round-robins across all 3 apiservers.

Why nftables DNAT? Why not use port 6443 directly?

▶

Why no HAProxy in front of Ingress Router?

▶

The Router IS Already HAProxy

Putting HAProxy in front of HAProxy = double proxy. Every app request would traverse two proxy hops, adding latency to all user-facing traffic. For API this is acceptable (low volume, latency-tolerant control plane). For app traffic serving thousands of requests/sec, it hurts.

✗ If HAProxy added in front

Client → VIP:443

↓

HAProxy (L4 LB)

↓

Router HAProxy (L7)

↓

app pod

Double proxy — extra latency on every request
Extra TCP connection setup & TLS overhead
Marginal gain — just picks which router
Router still does full L7 routing to pods

✓ Current setup (VIP direct)

Client → VIP:443

↓

Router HAProxy (L7)

↓

app pod

Single proxy hop — minimal latency
Simple — fewer failure points
Trade-off: only 1 router active (VIP)

Why API Gets HAProxy But Ingress Doesn't

Traffic type	API = control plane (low volume) vs Ingress = data plane (high volume)
Backend	kube-apiserver has no built-in LB. Router IS an LB — adding another is redundant
Double-proxy cost	API: acceptable (infrequent calls). Apps: every HTTP request pays the penalty
LB value	API: real gain (3 apiservers share load). Ingress: marginal (just picks which router does L7)

The Right Fix for Ingress HA on Bare Metal

Active/passive VIP is a known bare-metal limitation. Correct solutions eliminate proxy overhead entirely:

MetalLB + BGP/ECMP	Advertises VIP from multiple nodes. Upstream router distributes at L3. All 3 routers active. No extra proxy.
External LB	F5, HAProxy, or nginx in front. Distributes to all router nodes on 80/443 directly.
Sharded Ingress	Multiple IngressControllers with separate VIPs. Split traffic by domain.

40,000

KNI HAProxy maxconn (API)

50,000

Router HAProxy maxconn (Apps)

3 + 2

Masters + Workers

Floating VIPs

Click any component for detailed configuration

API traffic

DNAT

☁

Client

External request

☲

DNS + NAT

api.hj4h5...io → NAT → .201

API VIP

192.168.37.201:6443

on master2 (br-ex)

nftables DNAT

:6443 → :9445 (prerouting chain)

before userspace

KNI HAProxy

:9445 • TCP L4 • roundrobin

maxconn 40,000 • /readyz every 5s

Control Plane (3 masters)

kube-apiserver

master0 • .21:6443

GET /readyz ✓

kube-apiserver

master1 • .22:6443

GET /readyz ✓

kube-apiserver

master2 • .23:6443

GET /readyz ✓

API Request Path

Client → DNS → NAT → API VIP 192.168.37.201:6443 on master2. nftables rewrites 6443 → 9445. KNI HAProxy round-robins across all 3 apiservers with /readyz health checks.

App traffic

To pod

☁

Client

HTTPS request

☲

DNS + NAT

*.apps.hj4h5...io → NAT → .202

Ingress VIP

192.168.37.202:443

on master2 (br-ex)

master2 (VIP holder) — HostNetwork

Router HAProxy

HostNetwork :443 • L7 HTTP • 4 threads

maxconn 50,000 • balance random

Router

master0 (standby)

Router

master1 (standby)

Route / Service Match

Host header → Route → Service endpoints

TLS termination here

OVN-Kubernetes

Geneve overlay to pod IP

Worker / Master Nodes

pod/app-1

10.128.x.x:8080

pod/app-2

10.128.x.x:8080

pod/app-3

10.128.x.x:8080

App Request Path

Client → wildcard DNS → NAT → Ingress VIP 192.168.37.202:443. Router HAProxy via HostNetwork — no DNAT. L7 routing, TLS termination, OVN overlay to pods. Only 1 of 3 routers active (active/passive VIP).

Pod traffic (left)

kubelet traffic (right)

worker0 (192.168.37.24)

app pod

10.128.2.x

kubernetes.default.svc

ClusterIP :443 → :6443

OVN-Kubernetes LB

DNAT ClusterIP:443 → endpoint:6443

master2 (VIP holder)

kubelet

connects to VIP:6443

nftables redirect

output chain: VIP:6443 → :9445

KNI HAProxy

:9445 → roundrobin :6443

Control Plane

apiserver

master0 .21:6443

apiserver

master1 .22:6443

apiserver

master2 .23:6443

Two Different Internal Paths

Pod → apiserver (left): via kubernetes.default.svc:443, OVN LB DNATs to endpoint. Bypasses VIP/HAProxy.
kubelet on VIP holder (right): VIP:6443 → nftables redirect → :9445 → KNI HAProxy → all 3 apiservers.

External DNS

Internal DNS

☁

External Client

Public DNS

redhatworkshops.io nameservers

NAT Gateway

Public IP → internal VIPs

API VIP

192.168.37.201

Ingress VIP

192.168.37.202

Cluster Pod

10.128.x.x

Cluster CoreDNS

openshift-dns • ClusterIP :53

Node CoreDNS (KNI)

openshift-kni-infra • :53 per node

svc.cluster.local

myapp.ns.svc.cluster.local

api-int record

api-int.hj4h5...io → .201

DNS — External vs Internal

External: Public DNS → NAT gateway → internal VIPs.
Internal: Pods use cluster CoreDNS for svc.cluster.local. Node CoreDNS (KNI) resolves api-int and *.apps to VIPs.

API VRRP (ID 206)

Ingress VRRP (ID 45)

API VIP

192.168.37.201

VRRP ID 206

Ingress VIP

192.168.37.202

VRRP ID 45

Masters — API + Ingress VRRP

keepalived (master0)

192.168.37.21 • BACKUP

keepalived (master1)

192.168.37.22 • BACKUP

keepalived (master2)

192.168.37.23 • MASTER (both VIPs)

API Health Checks

chk_ocp_lb (+20) • chk_ocp_both (+5) • chk_mcs (+3)

Ingress Health Checks

chk_ingress (wt=0, FAULT) • chk_ingress_ready (+10) • chk_default_ingress (+50)

Workers — Ingress VRRP only

keepalived (worker0)

192.168.37.24 • FAULT (no router on :1936)

keepalived (worker1)

192.168.37.25 • FAULT (no router on :1936)

unicast VRRP adverts (1s on br-ex)

VRRP Topology

API VRRP (ID 206): 3 masters only. Priority 40 + weights (max 68).
Ingress VRRP (ID 45): All 5 nodes, but chk_ingress (weight=0) → FAULT on workers. Workers can never hold VIP.

Failure

Recovery

T+0s — Before Failure

Control Plane

master0

.21 • BACKUP

master1

.22 • BACKUP

master2 ●

.23 • MASTER

API VIP + Ingress VIP

T+0s — master2 Fails

master2 ✗

unreachable • VIPs orphaned

T+3s — VRRP Election

master0

API: 68

Ingress: 80

master1

API: 68

Ingress: 80

workers

Ingress: FAULT

T+3.1s — New Master + GARP

master0 ● (new MASTER)

API VIP .201 + Ingress VIP .202

GARP → switches update MAC

Already Running on master0

✓ KNI HAProxy (:9445)
✓ Router HAProxy (:80/:443)
✓ nftables rules
✓ kube-apiserver (:6443)

T+18s — HAProxy Backend Update

KNI HAProxy on master0

✓ master0 .21:6443 UP
✓ master1 .22:6443 UP
✗ master2 .23:6443 DOWN

Recovery Timeline

T+0s master2 down
T+3s VRRP election
T+3.1s GARP → traffic resumes
T+18s backend removed
T+60s GARP refresh

Failover

~3-5 seconds downtime. VRRP detects failure (3 missed adverts). New master GARPs. All components already running on standby. HAProxy removes dead backend after ~15s.

keepalived	Detects node-level failure → moves VIP to another node
HAProxy	Detects apiserver-level failure → stops sending traffic to unhealthy backend