Capture every decision for a customer deployment in one place. Export as JSON for the runbook or installer inputs.
OpenShift 4.20+ Architecture Design Workshop
Capture every decision for a customer deployment in one place. Everything you type is saved to your browser's localStorage and can be exported as JSON for the runbook or installer inputs. Use the cluster tabs below to design multiple clusters (e.g. management + hosted, prod + DR).
1. Cluster & Network
Full cluster domain = <name>.<base>
Network CIDRs
Machine Network is the only network your infrastructure team provisions (physical/VLAN). Cluster Network and Service Network are virtual overlays managed by OpenShift. EgressIP and MetalLB ranges are reserved from the Machine Network β they must not overlap with node IPs, VIPs, DHCP, or each other.
Subnet of node NICs
Default 10.128.0.0/14
/23 β 510 pod IPs/node
Default 250 β kubelet maxPods limit; constrained by Host Prefix IP capacity
Geneve overhead ~100 bytes
MTU in mixed environments (VM + Bare Metal): All nodes must have the same MTU on their primary interface β control-plane VMs and bare-metal workers alike. OVN-Kubernetes auto-detects the lowest MTU across nodes; a mismatch causes silent packet drops.
Jumbo frames (9000) checklist:
vSwitch / Distributed vSwitch on ESXi hosts → MTU 9000
VM port group carrying OCP traffic → MTU 9000
Every physical switch & router between VM hosts and bare-metal workers → MTU 9000
Bare-metal NIC, bond, and VLAN interfaces → MTU 9000 (see NMState section below)
Overlay overhead: OVN Geneve subtracts 100 bytes → set cluster network MTU to physical MTU − 100 (e.g., 9000 → 8900). With IPsec, subtract an additional 46 bytes. Best practice: If end-to-end jumbo frame support cannot be confirmed across all links, use MTU 1500 (overlay 1400). Post-install change is disruptive: MTU migration after install triggers two rolling reboots of all nodes — plan MTU at install time.
Reserved IP Ranges (from Machine Network)
These ranges are carved out of the Machine Network and must not overlap with node IPs, VIPs, or each other. Document them here so the network team can reserve them.
IPs assigned to EgressIP CRs for SNAT β must be routable from Machine Network, not used by nodes or DHCP
IPs for LoadBalancer Services (L2 or BGP mode) β must not overlap with node IPs or EgressIP range
Reserved / conflicting CIDRs β do NOT use these for Cluster or Service networks:
CIDR
Reason
10.88.0.0/16
Default Podman / CNI bridge network on RHCOS. Podman uses this for rootless container networking on nodes β overlapping will break host-level container operations and machine-config-daemon pivot.
100.64.0.0/16
Reserved by OVN-Kubernetes for internal transit switch (join subnet). Using it causes routing conflicts and pod connectivity failures.
100.88.0.0/16
Reserved by OVN-Kubernetes for internal masquerade / gateway operations (egress SNAT). Configurable via v4InternalMasqueradeSubnet but defaults to this range.
169.254.0.0/16
Link-local range (RFC 3927). Used by cloud metadata services and APIPA β will collide with node-level networking.
224.0.0.0/4
Multicast range β not routable for unicast pod traffic.
255.255.255.255/32
Broadcast address.
Also avoid overlap with your Machine Network, corporate VPNs, on-prem data-center ranges, or any network reachable via the node's default gateway. The installer does not validate against external routing β only against internal overlap between Cluster, Service, and Machine CIDRs.
VIPs, DNS, NTP
api.<cluster>.<base>
*.apps.<cluster>.<base>
Must resolve api, api-int, *.apps
Proxy Configuration
The installer auto-adds node, pod, and service CIDRs
Required proxy allowlist for OpenShift installation
If the cluster goes through a forward proxy, the following destinations must be allowed (HTTPS/443 unless noted). Block any of these and the install or day-2 operations will fail silently or hang.
Destination
Purpose
quay.io
Primary container image registry for OpenShift release and operator images
cdn.quay.io
CDN layer for Quay β serves image blobs
cdn01.quay.io β cdn03.quay.io
Additional Quay CDN endpoints (may expand over time)
registry.redhat.io
Red Hat certified operator and container images
registry.access.redhat.com
Legacy Red Hat container image registry (still referenced by some images)
access.redhat.com
Red Hat Errata and content (signature verification)
sso.redhat.com
Red Hat SSO β token auth for registry pulls and console.redhat.com
Insights / Telemetry (legacy endpoint, still in use)
no_proxy auto-populated by installer:.cluster.local, .svc, localhost, 127.0.0.1, API VIP, Ingress VIP, Machine CIDR, Pod CIDR, Service CIDR. Add any internal registries, Git repos, artifact stores, or corporate services that must bypass the proxy.
Air-gapped / disconnected installs: If you cannot allow-list the above, you must mirror all content to a local registry using oc mirror or oc adm release mirror. The proxy section becomes N/A β instead, configure ImageDigestMirrorSet / ImageTagMirrorSet CRs (OCP 4.14+; the older imageContentSources field is deprecated).
Load Balancer
External = you manage a separate LB appliance/VM. Built-in = OpenShift deploys HAProxy & Keepalived on nodes (IPI/ABI default).
Auto-filled from API VIP
Auto-filled from API VIP
Auto-filled from Ingress VIP
Built-in LB (default for IPI & ABI): OpenShift deploys HAProxy as a static pod on every control-plane node (and workers with router pods for Ingress). Keepalived manages the API and Ingress VIPs via VRRP (Virtual Router Redundancy Protocol). No external load balancer needed.
How VRRP Failover Works
Keepalived elects a MASTER node for each VIP via VRRP priority & advertisements (multicast 224.0.0.18)
MASTER sends periodic VRRP advertisements (default every 1 s)
If BACKUP nodes miss 3 consecutive advertisements (~3 s), the highest-priority BACKUP promotes itself
New MASTER sends gratuitous ARP to update upstream switch MAC tables
API & Ingress VIPs fail over independently β they can land on different nodes
Failover time: ~3-5 seconds (VRRP dead interval + gratuitous ARP propagation). Existing TCP connections are dropped; clients must reconnect.
Comparison: External LB vs Built-in (HAProxy + Keepalived)
Aspect
Built-in (HAProxy + Keepalived)
External Load Balancer
Setup
Zero-config β deployed automatically by installer
Must provision and configure LB before install
Cost
Free (part of OCP)
Depends on vendor (F5, AVI, NSX ALB can be expensive)
Requires L2 adjacency between nodes β VRRP uses multicast on the same broadcast domain
Works across L3 subnets and data centers
Failover speed
~3-5 s (VRRP advertisement timeout + ARP)
Sub-second to seconds depending on vendor and health-check interval
Health checks
Basic: Keepalived monitors HAProxy process and VIP interface
Rich: HTTP/TCP health checks, connection draining, slow-start
TLS termination
Passthrough only (HAProxy L4 mode for API; router handles TLS for apps)
Can terminate and re-encrypt, offload TLS, insert headers
Scalability
Good for small/mid clusters. All traffic funnels through one active VIP holder
Better for large clusters β can spread across multiple LB nodes, use ECMP
Observability
Limited β HAProxy stats via static pod logs, no dashboard
Full vendor dashboards, connection metrics, latency tracking
Multi-site / DR
Not supported β L2-bound VIP cannot span sites
GSLB, DNS-based failover, cross-DC routing
Customization
Minimal β HAProxy config managed by MCO, manual edits get overwritten
Full control: persistence, rate limiting, WAF, iRules, etc.
Switching after install is disruptive. Going from built-in to external requires re-configuring VIPs, DNS, and updating the InfrastructureStatus resource. Plan this choice before day 1.
API & Machine Config Server
*.apps Ingress (Routes) β targets workers with router pods
Diagrams auto-update from the node table below. Nodes with Role containing master appear in the API diagram. *.apps traffic goes only to nodes with the Router checkbox enabled. When Role = master/worker, both diagrams combine into one. By default, router pods run as a DaemonSet on all workers β restrict with IngressController.spec.nodePlacement.
Nodes (Hardware, IP, VLAN, FQDN)
#
Role
Hostname (FQDN)
CPU
RAM
Disk
RAID
IP
Subnet Mask
VLAN
Gateway
Router
OCP Node
DNS
Required DNS Records
All records below must resolve before installation begins. The installer validates DNS as a preflight check. Records auto-populate from cluster name, base domain, VIPs, and node entries above.
Record (FQDN)
Type
Target
Purpose
Wildcard DNS: The *.apps record must be a true wildcard A record, not a CNAME. Some DNS providers (e.g. certain Windows DNS or split-horizon setups) handle wildcards differently β test with dig test123.apps.<cluster>.<base> before install.
Firewall Rules
Rules auto-populate from cluster, VIP, node, DNS, NTP, proxy, and SMTP fields above. Share this table with the network/security team before installation. Direction is source β destination.
How to represent OCP node sources in firewall rules
Source IP that control plane nodes see from the API load balancer
Source IP that router nodes see from the Ingress load balancer
Category
Source
Destination
Port(s)
Description
2. Platform & Installer
ABI = air-gapped; Assisted = SaaS
NMState / MachineConfig bond interface name
Node BMC Addresses
#
Hostname
BMC Address
Never store BMC passwords in this file. Use a secret manager (Vault / Sealed Secrets) and reference them in install-config.yaml via the bmc.password field at install time.
Port Mapping (NICs / Switch Ports)
Network interface stack per bare-metal host: physical NICs are bonded, then VLANs are tagged on the bond with IP addresses assigned. Each row is one interface β add multiple rows per host to build the full stack.
#
Hostname
Interface
Type
MAC Address
Bond / Parent
Switch Port
VLAN ID
MTU
IP / Prefix
Gateway
Purpose
Tip: For a typical bonded host, add rows in order: 2Γ ethernet (bond slaves with MACs & switch ports) β 1Γ bond (bond0, mode 802.3ad) β 1Γ vlan (bond0.100 with IP). Use "Preset" to auto-populate this pattern.
vSphere Connection
Version requirements (OCP 4.20): vSphere / vCenter 8.0 Update 1 or later (or VMware Cloud Foundation 5+). vSphere 7.x is no longer supported. VM hardware version 15 minimum. The vSphere CSI Driver Operator has the same 8.0 U1+ requirement.
UPI only β IPI uploads its own OVA automatically
OCP requires version 15 minimum (RHCOS OVA default)
IPI requires DHCP unless static IPs configured via hosts: in install-config
Logical name in failureDomains:
vCenter tag category openshift-region β single-FD: any label
vCenter tag category openshift-zone β single-FD: any label
Never store vCenter passwords in this workshop export. Use a secret manager. The password lives only in install-config.yaml (entered at the openshift-install create install-config wizard prompt) and post-install in the vsphere-creds secret in kube-system. (GOVC_PASSWORD is for the govc CLI only β the installer does not read it.)
Design checklist:
Time sync: disable VMware Tools periodic time sync on OCP VMs β chrony (NTP servers above) must own the clock.
etcd latency: control-plane datastore needs <10 ms fsync (p99). Watch shared vSAN/NFS under contention; use eager-zeroed thick or a dedicated datastore for etcd-heavy clusters.
DRS anti-affinity: control-plane VMs must not share an ESXi host. IPI creates VM-VM anti-affinity rules automatically (4.13+); UPI β create the rule manually. Minimum 3 ESXi hosts for HA control plane.
disk.EnableUUID = TRUE: required on every node VM for the vSphere CSI driver. IPI sets it automatically; UPI must set it in each VM's advanced config.
Scope: Apply permissions at the vCenter level and propagate to children. For UPI, fewer permissions are needed (no VM creation automation).
Abbreviated list. Do not build the service-account role from this table alone β installer validation will fail on missing privileges. Copy the full per-object list from the OCP 4.20 docs: Installing on vSphere β Required vCenter account privileges.
The flat platform.vsphere.vcenter/username/datacenter/defaultDatastore/cluster fields are deprecated since 4.13 β the installer warns and converts them. Use vcenters: + failureDomains: as above. Multi-AZ: add one failure domain per vSphere cluster, with real region/zone tags attached in vCenter. Values auto-fill from the fields above.
UPI: VM Provisioning Requirements
Clone each node VM from the RHCOS OVA template named above (do not boot the template itself).
Set disk.EnableUUID = TRUE in each VM's advanced configuration parameters β mandatory for vSphere CSI.
Inject ignition via vApp/guestinfo properties per VM:
guestinfo.ignition.config.data = base64-encoded ignition file (bootstrap: pointer to bootstrap.ign on HTTP server; masters: master.ign; workers: worker.ign)
Default retention: 15 days. Runs 2 replicas in openshift-monitoring. Formula: disk β retention_s Γ samples/s Γ bytes/sample.
Architecture: The Cluster Monitoring Operator deploys Prometheus (2 HA replicas), Alertmanager, and Thanos Querier in openshift-monitoring. Each Prometheus has a Thanos Sidecar that exposes a StoreAPI β Thanos Querier is the unified PromQL endpoint that aggregates data from all sidecars. The OCP Console queries Thanos Querier, not Prometheus directly. Alertmanager receives firing alerts and routes to receivers. When User Workload Monitoring is enabled, a separate Prometheus + Thanos Sidecar + Thanos Ruler is deployed in openshift-user-workload-monitoring. Thanos Ruler evaluates recording and alerting rules for user workloads, querying data via Thanos Querier and sending alerts to Alertmanager.
Sizing parameter reference
Parameter
Description
Retention (days)
How long Prometheus keeps TSDB data on disk. Default in OpenShift: 15 days (15d). Longer retention = more disk. Set via prometheusK8s.retention in the cluster-monitoring-config ConfigMap. For long-term storage beyond the retention window, use prometheusK8s.remoteWrite to an external system (Thanos Receive, Mimir, an observability platform) β the bundled Thanos sidecar only serves in-cluster queries and cannot be attached to object storage.
Total Nodes
Number of nodes in the cluster. Each node runs a node-exporter DaemonSet pod that emits hardware/OS metrics. More nodes = more time series = more disk and memory. Include all roles (master, worker, infra).
Samples/sec per Node
Average metric samples ingested per second per node. Baseline for cluster-monitoring stack is ~8,000β12,000 samples/sec/node. This includes kubelet, cAdvisor, node-exporter, kube-state-metrics, and etcd metrics. Enabling User Workload Monitoring (UWM) or adding ServiceMonitors increases this number β measure with prometheus_tsdb_head_samples_appended_total.
Bytes/Sample
Average on-disk size per sample after TSDB compression. Typical range: 1.5β2 bytes. This is the post-compaction size β TSDB blocks compress efficiently. The default 2 bytes is a conservative estimate; low-cardinality clusters may see closer to 1.5.
Replicas
Number of Prometheus replicas. Default: 2 for HA in openshift-monitoring. Each replica stores a full copy of all metrics independently (no shared storage). Total storage = per-replica size Γ replicas. Do not reduce below 2 in production β you lose HA during rolling updates.
Headroom %
Extra buffer on top of the calculated disk size. Accounts for: WAL (Write-Ahead Log) segments (~20% of retention volume), compaction temporary files (TSDB merges blocks in place), series churn from pod restarts and deployments, and growth margin. Default 40% is suitable for most production clusters. Reduce to 20β30% if the cluster is stable with low churn.
Cluster-monitoring baseline: ~8β12k/node
Compressed TSDB β 1.5β2 B
Churn + compaction + WAL
Configure via the cluster-monitoring-config ConfigMap β set prometheusK8s.retention and volumeClaimTemplate (PVC).
User Workload Monitoring (UWM)
UWM runs a separate Prometheus in openshift-user-workload-monitoring to scrape user-defined ServiceMonitor and PodMonitor CRs. Sized independently from platform monitoring β has its own retention and PVC settings via the user-workload-monitoring-config ConfigMap.
Set enableUserWorkload: true in cluster-monitoring-config
Default 15d β set via user-workload-monitoring-config
Resource impact: UWM adds 2 Prometheus pods + 1 Thanos Ruler pod. Each Prometheus replica needs its own PVC. Plan for ~2 vCPU and 6β8 GB RAM per replica as baseline β scales with number of custom metrics and scrape targets.
Alertmanager β SMTP / Email Notifications
Alertmanager is configured via a Secret named alertmanager-main in openshift-monitoring. SMTP settings go in the global section of the Alertmanager config.
host:port β use 587 for STARTTLS (Prometheus Alertmanager does not support implicit TLS on port 465)
Leave blank if relay does not require auth
Never store SMTP passwords in the workshop export. Store them in a Secret and reference via Alertmanager config's smtp_auth_password_file or use a no-auth internal relay.
4. Storage β CSI & Protocol
Storage provisioning and CSI drivers are typically outside Red Hat's support scope β the storage vendor provides and supports the CSI driver. Document the storage backend here for the implementation runbook.
Support boundary: Red Hat supports the CSI interface and StorageClass API. The CSI driver itself, storage array configuration, zoning, LUN masking, and protocol-level troubleshooting are the storage vendor's responsibility. Ensure the vendor's CSI driver is certified for OpenShift β check the Red Hat Ecosystem Catalog.
CSI driver name β must be certified for OCP
Design checklist:
Multipath: iSCSI / FC / NVMe-oF need multipathd enabled on nodes via a day-1 MachineConfig (and iSCSI initiator config). Plan two independent fabric/network paths.
Default StorageClass: exactly one SC should carry storageclass.kubernetes.io/is-default-class: "true" β multiple defaults break PVC binding for claims without an explicit class.
Capabilities: confirm the CSI driver supports volume expansion (allowVolumeExpansion), snapshots (create a VolumeSnapshotClass), and cloning β needed by OpenShift Virtualization and backup tooling (OADP).
Access modes: NFS / CephFS = RWX filesystem; iSCSI / FC / RBD block = RWO (RWX only as raw block). Workloads needing shared writable volumes (e.g. some CI, AI training) need an RWX-capable class.
CSI Architecture β How Persistent Volumes Are Provisioned
5. Logging β Loki (LokiStack)
OCP 4.20 uses Loki via the Cluster Logging Operator (ClusterLogForwarder / LokiStack). Storage is object storage (S3, ODF, Swift, Azure, GCS). Sizing follows the t-shirt size model from Red Hat docs.
Architecture: The Logging Operator deploys Vector (DaemonSet) on every node to collect container and journal logs. ClusterLogForwarder CR defines pipelines β logs flow to the local LokiStack (managed by the Loki Operator) and/or external destinations. Loki stores compressed chunks in object storage. The Cluster Observability Operator installs the UIPlugin CR to enable log viewing in the OpenShift Console.
Sizing parameter reference
Parameter
Description
LokiStack Size
T-shirt size that determines component replica counts, resource requests/limits, and concurrency. Each size has a maximum ingestion rate: 1x.demo β€100 GB/day (no HA), 1x.pico β€50 GB/day, 1x.extra-small β€100 GB/day, 1x.small β€500 GB/day, 1x.medium β€2 TB/day.
Compression Factor
Ratio of compressed chunk size to raw log size. Loki compresses log data into chunks before writing to object storage. Typical values range from 0.06 to 0.20 (i.e. 6β20% of raw size). Lower values mean better compression. The default 0.15 is a safe middle estimate; actual ratio depends on log entropy and repetitiveness.
Replication Factor
Number of ingester replicas that receive each write. In OpenShift Logging 6.x (OCP 4.20+) this is automatically set to 2 for all production sizes and cannot be overridden in the CR: 1x.demo = none (no HA), 1x.pico = 2, 1x.extra-small = 2, 1x.small = 2, 1x.medium = 2.
This controls Loki's internal write replication across ingesters (WAL durability). It does not directly multiply object-store usage β object storage has its own replication (e.g. S3 11Γ9s durability, ODF 3-way replicated pool). Do not double-count: if your object store already replicates, storage sizing uses replication factor = 1.
Headroom %
Extra storage buffer beyond the calculated compressed volume. Accounts for:
β’ Index files β TSDB index (labels, series metadata) stored alongside chunks
β’ Compaction temp space β compactor rewrites/merges chunks in-place, needing temporary headroom
β’ WAL segments β Write-Ahead Log on ingester PVCs (~2 GB per ingester by default)
β’ Chunk overlap β during retention enforcement, chunks spanning the boundary are kept until they fully expire
β’ Growth margin β log volume tends to grow as workloads scale
Recommended: 20% for stable clusters, 30β40% for clusters with bursty log patterns or high pod churn.
Retention (per-tenant)
How many days of logs to keep, configured separately for each tenant (log type) in OCP 4.20+. OpenShift Loki has three built-in tenants:
β’ application β container logs from user workload namespaces
β’ infrastructure β logs from openshift-*, kube-* namespaces and node journals
β’ audit β Kubernetes API audit logs, OAuth audit, OVN audit
Configured via LokiStack.spec.limits.global.retention (default for all) and LokiStack.spec.limits.tenants.<name>.retention (per-tenant override). Loki's compactor deletes chunks older than retention during its compaction cycle. Typical patterns: keep audit logs longest (compliance), infrastructure medium, application shortest.
application tenant ingest rate
infrastructure tenant ingest rate
audit tenant ingest rate
Loki chunks ~6β20% of raw
Auto-set by LokiStack size
Estimation only: These numbers are rough estimates based on assumed ingest rates and compression ratios. Actual storage depends on log verbosity, application behavior, pod churn, and real-world compression. Use this as a starting point for capacity planning β monitor actual usage after deployment and adjust accordingly.
Per-tenant retention is configured in LokiStack.spec.limits.tenants.<name>.retention where name is application, infrastructure, or audit. The replication factor is fixed per LokiStack size and cannot be changed in the CR. Object storage sizing should not be multiplied by the replication factor β that controls ingester-level WAL replication, not object-store copies. If your object store has its own replication (ODF replicated pool, S3 durability), storage is already durable.
Log Forwarding Destinations (ClusterLogForwarder)
Use ClusterLogForwarder to send logs to external systems in addition to (or instead of) the local LokiStack. Multiple outputs and pipelines can be defined.
#
Output Name
Type
URL / Endpoint
Input Sources
Common output types: lokiStack (default local), elasticsearch, kafka, splunk, syslog, cloudwatch, http (generic webhook). Input sources: application, infrastructure, audit. TLS certs and secrets are referenced in the CR, not stored here.
6. Authentication β OAuth Identity Providers
OpenShift uses the built-in OAuth server. Identity providers are configured via the OAuth cluster resource (oc edit oauth cluster). Multiple providers can coexist.
OAuth Authentication Flow Diagram
HTPasswd
Simple file-based auth β good for break-glass admin accounts, lab clusters, or bootstrapping before LDAP is ready. The htpasswd file is stored in a Secret in openshift-config.
Shows on the login page
openshift-config namespace
Created with: htpasswd -c -B -b users.htpasswd <user> <pass>
HTPasswd setup commands
# Create the htpasswd file
htpasswd -c -B -b users.htpasswd admin 'CHANGE_ME'
htpasswd -B -b users.htpasswd emergency-admin 'CHANGE_ME'
# Create the secret in openshift-config
oc create secret generic htpasswd-secret \
--from-file=htpasswd=users.htpasswd \
-n openshift-config
# Patch the OAuth resource
oc edit oauth cluster
# Add under spec.identityProviders:
# - name: htpasswd-local
# type: HTPasswd
# htpasswd:
# fileData:
# name: htpasswd-secret
# Grant cluster-admin to the admin user
oc adm policy add-cluster-role-to-user cluster-admin admin
LDAP / Active Directory
Connects to an external LDAP directory (AD, FreeIPA, OpenLDAP). Users are looked up on each login β no sync is needed for authentication, but oc adm groups sync is required for RBAC via group membership.
Shows on the login page
ldap:// or ldaps:// Β· base DN Β· attribute Β· scope Β· filter
Service account for LDAP search (leave blank for anonymous bind)
Secret in openshift-config containing the bind password
ConfigMap in openshift-config with ca.crt (required for ldaps://)
LDAP Attribute Mapping
Unique, stable identifier (default: dn)
AD: sAMAccountName Β· LDAP: uid
Shown in the OCP console
LDAP Group Sync (for RBAC)
Only sync groups matching this filter
CronJob in openshift-authentication or external
LDAP group sync command
# Create a sync config file (ldap-sync.yaml):
kind: LDAPSyncConfig
apiVersion: v1
url: ldaps://ldap.example.com:636
bindDN: cn=ocp-svc,ou=ServiceAccounts,dc=example,dc=com
bindPassword:
file: /etc/secrets/bindPassword
ca: /etc/ldap-ca/ca.crt
insecure: false
activeDirectory:
usersQuery:
baseDN: "dc=example,dc=com"
scope: sub
derefAliases: never
filter: (objectClass=person)
pageSize: 0
userNameAttributes: [ sAMAccountName ]
groupMembershipAttributes: [ memberOf ]
# Run the sync
oc adm groups sync --sync-config=ldap-sync.yaml --confirm
# Bind a synced group to a cluster role
oc adm policy add-cluster-role-to-group cluster-admin ocp-admins
Never store LDAP bind passwords in this workshop export. Use oc create secret generic ldap-bind-password --from-literal=bindPassword='...' -n openshift-config and reference the secret name above.
OpenID Connect (OIDC)
Connects the OAuth server to an OIDC issuer (Microsoft Entra ID, Keycloak / RH-SSO, Okta, PingFederate). Uses the Authorization Code flow β the issuer must allow the cluster's redirect URI https://oauth-openshift.apps.<cluster>.<base>/oauth2callback/<provider-name>.
Shows on the login page β also part of the redirect URI
Must serve /.well-known/openid-configuration over https
Secret in openshift-config (never the secret value itself)
Entra ID: upn or email Β· Keycloak: preferred_username
Maps IdP groups to OCP groups on each login (no sync job needed)
openshift-config β only if the issuer uses a private CA
New in 4.20 (GA): Instead of the built-in OAuth server, you can enable direct external OIDC authentication β the kube-apiserver validates the IdP's tokens natively (upstream structured authentication config, oc gets an oc-oidc plugin with Auth Code + PKCE). Replaces OAuth entirely and can be switched back. Consider it when the customer requires IdP-controlled sessions or one IdP across many clusters. See Authentication β Enabling direct authentication with an external OIDC identity provider.
7. Certificates β Ingress, API & Custom CAs
By default, OpenShift generates self-signed certificates for the API server and the default Ingress Controller (*.apps). Browsers and CLI tools will show TLS warnings until you replace them with certificates signed by a trusted CA. You can replace them independently.
Planning note: Certificate replacement requires a Secret of type kubernetes.io/tls containing the full chain (cert + intermediates) and private key. Coordinate with the customer's PKI / security team β lead time for CA-signed certs can be days to weeks.
Certificate Chain & Placement Diagram
Default Ingress Certificate (*.apps)
Replaces the self-signed wildcard certificate used by the default IngressController for all Routes. Modern TLS clients validate SAN (Subject Alternative Name), not CN β the wildcard must appear in SAN.
Auto: *.apps.<cluster>.<base>
Auto: *.apps.<cluster>.<base> β must be in SAN, CN alone is not sufficient
Extra SANs beyond the wildcard
Replace default Ingress certificate
# 1. Create the TLS secret in openshift-ingress namespace
oc create secret tls apps-tls-cert \
--cert=wildcard-apps.crt \
--key=wildcard-apps.key \
-n openshift-ingress
# Note: wildcard-apps.crt must contain the full chain:
# - Server certificate (*.apps.ocp.example.com)
# - Intermediate CA(s)
# - (Optional) Root CA
# 2. Patch the IngressController to use the new certificate
oc patch ingresscontroller default \
-n openshift-ingress-operator \
--type=merge \
-p '{"spec":{"defaultCertificate":{"name":"apps-tls-cert"}}}'
# 3. Verify β the router pods will restart automatically
oc get pods -n openshift-ingress -w
# 4. Test with curl
curl -v https://console-openshift-console.apps.ocp.example.com 2>&1 | grep issuer
API Server Certificate
Replaces the self-signed certificate used by the Kubernetes API server (api.<cluster>.<base>:6443). After replacement, oc login and kubeconfig clients will trust the API without --insecure-skip-tls-verify.
Auto: api.<cluster>.<base>
Replace API server certificate
# 1. Create the TLS secret in openshift-config namespace
oc create secret tls api-tls-cert \
--cert=api-server.crt \
--key=api-server.key \
-n openshift-config
# Note: The certificate SAN must include:
# - api.<cluster>.<base> (e.g., api.ocp.example.com)
# The cert file should include the full chain (server + intermediates)
# 2. Patch the APIServer resource
oc patch apiserver cluster \
--type=merge \
-p '{"spec":{"servingCerts":{"namedCertificates":[{"names":["api.ocp.example.com"],"servingCertificate":{"name":"api-tls-cert"}}]}}}'
# 3. Monitor the kube-apiserver rollout (takes several minutes)
oc get co kube-apiserver -w
# 4. Verify
oc login https://api.ocp.example.com:6443 -u kubeadmin
# Should connect without TLS warnings if CA is trusted
Warning: API server certificate replacement triggers a rolling restart of kube-apiserver pods across all control plane nodes. This is non-disruptive for workloads but the API may be briefly unavailable during rollout (~5β10 minutes). Plan this during a maintenance window.
The installer-generated kubeconfig keeps working after replacement β it pins the internal CA, and the internal endpoints keep serving the internal cert. Use a separate certificate from the Ingress wildcard: api. must not be covered by *.apps., and reusing one cert for both couples two unrelated rotation lifecycles.
Additional Trusted CA Bundle
If your organization uses a private/internal CA (for proxy, registry, LDAP, etc.), add it to the cluster-wide trusted CA bundle so all components trust it. This is separate from the API/Ingress certs above.
Namespace: openshift-config
Add custom CA to cluster trust bundle
# 1. Create a ConfigMap with the CA bundle in openshift-config
oc create configmap user-ca-bundle \
--from-file=ca-bundle.crt=corp-root-ca.pem \
-n openshift-config
# Note: ca-bundle.crt can contain multiple PEM certificates concatenated
# 2. Patch the proxy/cluster resource to use the custom CA
oc patch proxy/cluster \
--type=merge \
-p '{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'
# This injects the CA into:
# - All pods via the trusted-ca-bundle ConfigMap (if they opt in via label)
# - Machine Config for node-level trust (RHCOS trust store)
# - Proxy-aware components (image pulls, egress)
# 3. Verify the CA is distributed
oc get configmap trusted-ca-bundle -n openshift-config-managed -o yaml | head -20
Pods opt in by adding the label config.openshift.io/inject-trusted-cabundle: "true" to a ConfigMap. The Cluster Network Operator (CNO) then injects the merged system + custom CAs. This is how builds, S2I, and image registry connections pick up your internal CA.
cert-manager Operator: For automated lifecycle (issuance, renewal, revocation), install the cert-manager Operator for Red Hat OpenShift from OperatorHub. It supports ACME (Let's Encrypt), Venafi, Vault, and private CA issuers. Configure ClusterIssuer and Certificate CRs to auto-manage both Ingress and API certs.
8. Hosted Control Plane (HCP) Configuration
Hosted Control Planes (HyperShift) decouple the control plane from the data plane. The control plane runs as pods on a management cluster, while worker nodes run on separate infrastructure. A single management cluster can host multiple hosted clusters β use the tabs below to design each one.
Prerequisites: The management cluster must have the multicluster-engine operator installed with the HyperShift add-on enabled. The management cluster needs sufficient capacity to host additional control plane pods (3 etcd + 3 API server + controllers per hosted cluster in HA mode).
Management Cluster
The OCP cluster hosting the control plane pods
Must be same or newer than hosted cluster version
Hosted Cluster Settings
Name of the HostedCluster CR β used to auto-create the CP namespace
Full domain = <cluster-name>.<base-domain> β used for API and *.apps
Namespace on mgmt cluster for HostedCluster CR
OCP version for the hosted cluster
Storage class on mgmt cluster for etcd PVCs
Pod network for the hosted cluster β must not overlap with mgmt cluster
Must not overlap with mgmt cluster CIDRs
IPs reserved for EgressIP CRs on hosted cluster β must be routable on worker network, not overlap with node IPs or VIPs
DNS servers for hosted cluster nodes β must resolve API and *.apps records
NTP servers for time synchronization on worker nodes β critical for certificate validation and log correlation
Subnet prefix length assigned to each node for pod IPs β default /23 (512 IPs per node)
Default 250 β kubelet maxPods limit; constrained by Host Prefix IP capacity
Control Plane Namespace: OpenShift automatically creates a namespace on the management cluster for the hosted control plane components. The namespace is named <namespace>-<cluster-name> (e.g. clusters-hosted-01). All control plane pods (etcd, kube-apiserver, kube-controller-manager, etc.) run here. You do not create it manually.
HCP Proxy Configuration
Proxy settings for the hosted cluster. These are applied to the HostedCluster CR and propagated to all worker nodes. The management cluster proxy config (if any) is separate β both must be configured independently.
Add internal registries, Git repos, artifact stores, or corporate services that must bypass proxy
Required if proxy performs TLS inspection β ConfigMap with ca-bundle.crt key in the hosting namespace
HCP proxy β critical no_proxy entries and gotchas
Required: 172.20.0.1 in no_proxy β HCP uses an internal Konnectivity tunnel between the control plane (on mgmt cluster) and kubelet/API on worker nodes. The 172.20.0.1 address is the in-cluster Kubernetes service IP used by the Konnectivity agent to reach the hosted kube-apiserver. If this is proxied, the tunnel breaks and nodes show NotReady with "unable to connect to kube-apiserver" errors. Always add 172.20.0.1 to no_proxy.
Loopback β various health checks and local endpoints
Hosted Cluster Service CIDR
Internal service IPs (e.g. 172.31.0.0/16) must not be proxied
Hosted Cluster Pod CIDR
Pod-to-pod traffic within the hosted cluster
Worker Node Machine CIDR
Node-to-node traffic on the data-plane network
Management Cluster API
Worker nodes register via mgmt cluster API β must bypass proxy if on same network
169.254.169.254
Cloud provider metadata service (AWS/Azure/GCP) β if applicable
HCP proxy differs from standalone OCP proxy:
Standalone OCP: proxy set in install-config.yaml, installer auto-populates no_proxy with node/pod/service CIDRs.
HCP: proxy set in HostedCluster CR under spec.configuration.proxy. No automatic no_proxy population β you must explicitly list all CIDRs and the 172.20.0.1 Konnectivity address.
Management cluster and hosted cluster proxy configs are independent. The mgmt cluster can use a different proxy (or none) from the hosted cluster.
If the mgmt cluster also uses a proxy, ensure the hosted cluster API endpoint (on mgmt side) is in the mgmt cluster's no_proxy.
Trust bundle: If proxy performs TLS interception (MITM), you must provide the proxy CA certificate as a ConfigMap in the hosting namespace and reference it in spec.configuration.proxy.trustedCA. Without this, image pulls and API calls from hosted cluster components will fail with "x509: certificate signed by unknown authority".
Authenticated proxy β URL-embedded credentials broken in HCP
Proxy URLs with credentials in the URL (e.g. http://proxyuser:P%40ssw0rd@172.16.27.40:3128) cause a Go URL parser error:
dial tcp: address proxyuser:P%40ssw0rd@172.16.27.40:3128: too many colons in address
This happens because the Go net/http proxy dialer receives the raw authority portion (user:pass@host:port) and sees multiple : separators β it fails to distinguish the password separator from the port separator. Percent-encoded characters like %40 (@) and special characters in the password make this worse but are not the only trigger β even simple passwords like 1q2w3e4r in the URL form http://proxyuser:1q2w3e4r@172.16.27.40:3128 can fail.
Workaround: Do not embed credentials in the proxy URL. Instead, configure the proxy to allow traffic from the cluster IP ranges without authentication (IP-based allowlist), or use a local unauthenticated proxy relay (e.g. a sidecar Squid instance that forwards to the authenticated upstream proxy).
This is a known limitation in Go's net/http stack that affects both standalone OpenShift and HCP. HCP is more impacted because the control plane pods on the management cluster also need proxy access, and there is no supported mechanism to inject proxy credentials separately from the URL.
If you must use an authenticated proxy, coordinate with the proxy team to whitelist the management cluster and worker node CIDRs for unauthenticated access.
https:// scheme for proxy URL β supported for httpsProxy only (OCP 4.20+)
The field names httpProxy / httpsProxy specify which traffic goes through the proxy (HTTP vs HTTPS), but the scheme in the URL controls how OpenShift connects to the proxy:
httpProxy β only http:// scheme allowed. Installer/API validation rejects https://.
httpsProxy β both http:// and https:// schemes accepted. Using https://proxy:3128 means OpenShift establishes a TLS connection to the proxy, then sends the CONNECT request through that encrypted channel.
Go 1.24 (used by OCP 4.20) has supported HTTPS proxy connections since Go 1.10. No runtime blocker.
Standalone OCP: Fully supported. Installer validation explicitly allows https scheme for the httpsProxy field.
HCP: Same API validation β technically supported, but less field-tested. Proxy runs through more components (Konnectivity, control plane pods on mgmt cluster), so test thoroughly.
If using https:// scheme, the proxy's TLS CA certificate must be in the trustedCA bundle β otherwise all HTTPS-proxied traffic fails with "x509: certificate signed by unknown authority".
Caveat: All Red Hat documentation examples use http:// for both fields. The https:// path is valid but uncommon β expect less community coverage and support familiarity.
Each NodePool CR defines a set of worker nodes. You can have multiple pools with different sizes, labels, or platforms.
#
Pool Name
Replicas
CPU
RAM
Root Disk
Auto Min
Auto Max
Labels / Notes
Agent Platform (Bare Metal) Settings
Worker nodes are provisioned via the Agent platform β bare-metal hosts are registered through InfraEnv and booted with the discovery ISO. Node dimensions (CPU, RAM, disk) come from the physical hardware β not from NodePool specs. The NodePool replicas field controls how many approved agents are consumed.
No NodePool sizing for Agent platform: Unlike KubeVirt or cloud platforms, Agent-based NodePools do not specify CPU/RAM/disk dimensions. The hardware spec is determined by the physical bare-metal hosts registered via InfraEnv and discovery ISO.
Assigned by MetalLB on the worker network β hosted cluster API endpoint
Assigned by external load balancer β hosted cluster *.apps endpoint
Network where agent-based worker nodes reside
Namespace with InfraEnv and approved Agent CRs on mgmt cluster
Agent Node Pools
For agent bare metal, each NodePool only needs a name and replica count. Hardware capacity varies per physical host β the replicas field controls how many approved agents are consumed from the inventory.
#
Pool Name
Replicas
Labels / Notes
Hosted Cluster Node Inventory
Hardware and network details for worker nodes in this hosted cluster. These are the bare-metal hosts that will be registered via InfraEnv and consumed by NodePool replicas. Node FQDNs auto-populate into the HCP DNS records table.
#
Role
Hostname (FQDN)
CPU
RAM
Disk
IP
Subnet Mask
VLAN
Gateway
Agent Host Port Mapping
Network interface stack per bare-metal host: physical NICs are bonded, then VLANs are tagged on the bond with IP addresses assigned. Each row is one interface β add multiple rows per host to build the full stack. Used by NMStateConfig and InfraEnv for discovery ISO boot.
#
Hostname
Interface
Type
MAC Address
Bond / Parent
Switch Port
VLAN ID
MTU
IP / Prefix
Gateway
Purpose
Tip: For a typical bonded host, add rows in order: 2Γ ethernet (bond slaves with MACs & switch ports) β 1Γ bond (bond0, mode 802.3ad) β 1Γ vlan (bond0.100 with IP). Use "Preset" to auto-populate this pattern.
NMStateConfig β Bonding & VLAN (InfraEnv)
When bare-metal worker nodes require bonded interfaces with LACP (802.3ad) and tagged VLANs, create an NMStateConfig resource per host. The InfraEnv references these configs so that the discovery ISO applies the correct network setup at boot time.
Prerequisites: The physical switch ports must be configured for LACP (802.3ad) channel groups and the target VLAN must be trunked to the host ports.
NMStateConfig YAML (bonding + VLAN)
apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
name: worker-01
namespace: hardware-inventory # must match InfraEnv namespace
labels:
infraenvs.agent-install.openshift.io: <infraenv-name>
spec:
# spec.interfaces β maps MAC addresses to interface names
# Used by the agent to identify which physical NIC is which
interfaces:
- name: ens1f0
macAddress: "aa:bb:cc:dd:ee:01"
- name: ens1f1
macAddress: "aa:bb:cc:dd:ee:02"
# spec.config β nmstate desired-state applied at boot
config:
interfaces:
- name: ens1f0
type: ethernet
state: up
mtu: 9000
- name: ens1f1
type: ethernet
state: up
mtu: 9000
- name: bond0
type: bond
state: up
mtu: 9000
link-aggregation:
mode: 802.3ad
options:
lacp_rate: fast
miimon: "100"
xmit_hash_policy: layer3+4
port:
- ens1f0
- ens1f1
- name: bond0.100
type: vlan
state: up
mtu: 9000
vlan:
base-iface: bond0
id: 100
ipv4:
enabled: true
address:
- ip: 10.0.100.11
prefix-length: 24
dhcp: false
ipv6:
enabled: false
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 10.0.100.1
next-hop-interface: bond0.100
dns-resolver:
config:
server:
- 10.0.0.5
- 10.0.0.6
search:
- example.com
InfraEnv referencing NMStateConfig
apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
name: hosted-01-infraenv
namespace: hardware-inventory
spec:
clusterRef:
name: hosted-01
namespace: clusters
pullSecretRef:
name: pull-secret
sshAuthorizedKey: "ssh-rsa AAAA..."
nmStateConfigLabelSelector:
matchLabels:
infraenvs.agent-install.openshift.io: hosted-01-infraenv
# Each NMStateConfig with the matching label will be embedded
# into the discovery ISO β one config per host MAC/interface
LACP timing: Use lacp_rate: fast (every 1s) for quicker failover detection. Some switches default to slow (30s). Ensure both sides match. Set miimon: 100 for 100ms link monitoring.
MTU consistency: The MTU must be configured end-to-end β physical NICs, bond, VLAN interface, and all upstream switch ports. In mixed environments, also verify that vSwitch / Distributed vSwitch and VM port groups on ESXi hosts match the bare-metal MTU. Mismatched MTU causes silent packet drops and degraded performance.
Agent-based HCP create command
# Create a hosted cluster with agent platform
hcp create cluster agent \
--name=hosted-01 \
--base-domain=example.com \
--api-server-address=api.hosted-01.example.com \
--etcd-storage-class=ocs-storagecluster-ceph-rbd \
--ssh-key=/path/to/ssh-key.pub \
--namespace=clusters \
--control-plane-availability-policy=HighlyAvailable \
--release-image=quay.io/openshift-release-dev/ocp-release:4.20.x-x86_64 \
--pull-secret=/path/to/pull-secret.json
# Control plane namespace is auto-created: clusters-hosted-01
# It contains etcd, kube-apiserver, kube-controller-manager pods
# Scale the node pool (replicas = number of agents to consume)
oc -n clusters scale nodepool hosted-01 --replicas=3
# Approve agents (after booting BM hosts with discovery ISO)
oc -n hardware-inventory get agents
oc -n hardware-inventory patch agent <agent-id> \
--type=merge -p '{"spec":{"approved":true}}'
# API VIP is assigned by MetalLB (configure MetalLB on worker network)
# Ingress VIP is assigned by external load balancer
KubeVirt Platform Settings
Worker nodes run as KubeVirt VMs on the management cluster. Requires OpenShift Virtualization (CNV) installed on the management cluster.
Ingress path: KubeVirt-based hosted clusters use NodePort as the backend for ingress. The external load balancer points *.apps traffic on port 443 to NodePort services on the cluster where the VMs reside (which may be the management cluster or a separate infrastructure cluster), which forward to the router pods running inside the KubeVirt VM workers.
Monitoring: Use oc get hostedcluster -n clusters on the management cluster to check status. Hosted cluster kubeconfig: hcp create kubeconfig --name=<cluster> --namespace=clusters > kubeconfig.
HCP Required DNS Records
Auto-generated from HCP cluster name, base domain, and VIP fields above. All records must resolve before the hosted cluster is created.
Record (FQDN)
Type
Target
Purpose
Wildcard DNS: The *.apps record for the hosted cluster must be a true wildcard A record pointing at the Ingress VIP, not a CNAME.
HCP Firewall Rules
Auto-generated from HCP fields above. Covers management β hosted cluster traffic, Konnectivity tunnel, API/Ingress endpoints, and agent discovery.
How to represent hosted worker sources in firewall rules