Nodes & Node Types

Understanding the physical and logical building blocks of an OpenShift cluster.

What is a Node?

A node is a physical or virtual machine that participates in the cluster. Every Linux node runs kubelet, CRI-O, and cluster networking components. RHCOS nodes also run the Machine Config Daemon; supported RHEL compute nodes use a separate operating-system management workflow.

• kubelet — agent ensuring containers run in pods
• CRI-O — lightweight OCI-compliant container runtime
• OVN-Kubernetes — SDN for pod networking
• Machine Config Daemon — applies OS-level configuration on RHCOS nodes

Node Architecture

Node Comparison

Inside Every Node — Component Deep Dive

OpenShift nodes run a common set of Kubernetes and platform services, with some host-management components specific to RHCOS. Understanding what each does is essential for troubleshooting and architecture decisions.

kubelet — The Node Agent

The kubelet is the primary agent on every node. It registers the node with the API server, watches for pod assignments, and ensures containers are running as expected.

• Pod lifecycle management — Creates, starts, stops, and restarts containers via CRI-O
• Image pulling — Pulls container images from registries based on imagePullPolicy
• Volume mounting — Coordinates with CSI drivers to stage and mount persistent volumes
• Health monitoring — Executes liveness, readiness, and startup probes. Restarts containers on liveness failure
• Resource enforcement — Enforces CPU/memory limits via cgroups. Evicts pods when node is under pressure (DiskPressure, MemoryPressure)
• Status reporting — Reports node conditions (Ready, MemoryPressure, DiskPressure, PIDPressure) and resource capacity to API server
• Static pods — On control plane nodes, kubelet manages static pods (etcd, kube-apiserver, kube-controller-manager, kube-scheduler) from /etc/kubernetes/manifests/

Copy# Check kubelet status on a node
oc debug node/<node-name> -- chroot /host systemctl status kubelet

# View kubelet logs
oc debug node/<node-name> -- chroot /host journalctl -u kubelet --no-pager -n 50

# Check node conditions
oc describe node <node-name> | grep -A5 Conditions

CRI-O — Container Runtime

CRI-O is the lightweight, OCI-compliant container runtime used by OpenShift. It implements the Kubernetes Container Runtime Interface (CRI) and completely replaces Docker.

• CRI gRPC interface — kubelet communicates with CRI-O via a Unix socket (/var/run/crio/crio.sock). CRI-O handles RunPodSandbox, CreateContainer, StartContainer, StopContainer, RemoveContainer
• Image management — Pulls images from registries, stores in local storage. Supports image signing and mirror registries
• Container execution — Uses runc (or crun) as the low-level OCI runtime to create Linux containers with namespaces, cgroups, and seccomp
• Version tracking — CRI-O version matches the Kubernetes version in each OCP release (e.g., OCP 4.14 = K8s 1.27 = CRI-O 1.27)
• No daemon — Unlike Docker, CRI-O has no long-running background daemon. Each container is a child of CRI-O managed by systemd
• Managed by MCO — CRI-O configuration (registries, runtimes, pids-limit) is delivered via MachineConfig objects. Never edit /etc/crio/ directly

Copy# Check CRI-O status
oc debug node/<node-name> -- chroot /host systemctl status crio

# List containers on a node via crictl
oc debug node/<node-name> -- chroot /host crictl ps

# Inspect a container
oc debug node/<node-name> -- chroot /host crictl inspect <container-id>

OVN-Kubernetes — Software-Defined Networking

OVN-Kubernetes is the default CNI plugin since OpenShift 4.12. It provides pod networking, service load balancing, and network policy enforcement using Open Virtual Network (OVN) on top of Open vSwitch (OVS).

• Geneve tunnels — Encapsulates pod-to-pod traffic across nodes in Geneve (Generic Network Virtualization Encapsulation) tunnels instead of VXLAN
• Replaces kube-proxy — Service load balancing is done natively in OVN via OVS flows. No iptables rules needed
• Logical topology — Creates logical switches (one per node), logical routers, ports, ACLs (for NetworkPolicy), and load balancers
• NetworkPolicy enforcement — Implemented as OVN ACLs. Stateful firewall applied at the OVS level. Evaluates ingress and egress rules
• EgressIP — Assigns a static source IP to pods in a namespace for outbound traffic. Useful for firewall allow-listing
• Multus — CNI meta-plugin allowing pods to have additional network interfaces (SR-IOV, Macvlan, bridge) alongside the primary OVN interface

CoreDNS — Cluster DNS

CoreDNS is the cluster DNS server providing service discovery. Every pod is configured to use CoreDNS for name resolution.

• Service discovery — Resolves <service>.<namespace>.svc.cluster.local to the Service ClusterIP
• Headless Services — For Services with clusterIP: None, CoreDNS returns individual pod IPs (A records for StatefulSet pods: pod-0.svc.ns.svc.cluster.local)
• External DNS — Forwards queries for non-cluster domains to upstream DNS servers configured on the node
• Managed by DNS Operator — The DNS Operator (openshift-dns-operator namespace) manages CoreDNS pods running as a DaemonSet in the openshift-dns namespace
• Port 53 — Listens on UDP/TCP port 53. When you restrict egress with NetworkPolicy, you must allow port 53 to openshift-dns or DNS resolution breaks

Machine Config Daemon (MCD)

The MCD is the node-level agent of the Machine Config Operator (MCO). It runs on RHCOS nodes and handles supported OS-level configuration delivery. RHEL compute nodes are maintained outside this RHCOS/MCO lifecycle.

• Watches rendered MachineConfig — Each MachineConfigPool (master, worker, infra) has a rendered config. MCD compares current node config with the target
• Delivers configuration — Writes files (e.g., chrony.conf, registries.conf), enables/disables systemd units, sets kernel arguments, manages SSH authorized keys
• Triggers reboot — Most config changes require a node reboot. MCD coordinates: cordon → drain → apply → reboot → uncordon. One node at a time per pool
• Avoid ad hoc SSH changes — Use MachineConfig and supported Operators for persistent RHCOS configuration. Administrators still review changes, monitor rollouts, troubleshoot degraded pools, and maintain recovery access

Node Tuning Operator

The Node Tuning Operator manages node-level performance tuning via Tuned profiles. A tuned DaemonSet pod runs on every node applying sysctl parameters, hugepage allocation, CPU governor settings, and NUMA pinning.

• Default profiles — Ships with profiles for general workloads. Custom profiles for real-time, low-latency, and high-performance computing
• PerformanceProfile CRD — For telco/edge workloads, defines CPU isolation, hugepages, real-time kernel, and NUMA topology
• No reboot needed — Most tuning changes are applied live (unlike MachineConfig changes). Kernel parameters take effect immediately

CSI Node Plugin

Each storage provider runs a CSI Node Plugin as a DaemonSet on every node. It handles the last-mile operations of making storage available to pods.

• NodeStageVolume — Formats the device (if needed) and mounts it to a staging path on the node
• NodePublishVolume — Bind-mounts from the staging path into the pod's mount namespace. The pod sees a regular directory
• Driver registrar — Registers the CSI driver with kubelet so it knows which driver handles which StorageClass
• Multiple drivers — A node can run multiple CSI node plugins simultaneously (e.g., ODF for Ceph + vSphere for VMDKs)

Monitoring & Logging Agents

Every node runs DaemonSet pods for observability:

• node-exporter — Prometheus exporter that collects hardware and OS metrics: CPU usage, memory, disk I/O, filesystem space, network stats. Scraped by platform Prometheus
• Vector (or Fluentd) — Log collector DaemonSet that reads container logs from /var/log/containers/ and journal logs, then forwards to Loki or external log stores
• Collector (ACS) — If ACS is installed, CO-RE BPF-based DaemonSet monitors process execution, network connections, and file access on every node for runtime threat detection

Taints, Tolerations & Affinity

How the scheduler decides which node runs each pod — and how you control placement.

Scheduling Decision Flow

Taints & Tolerations

A taint on a node repels pods. A toleration on a pod allows it to schedule on a tainted node. Taints are key=value:effect.

Copy# Add taint to a node
oc adm taint nodes worker-1 dedicated=infra:NoSchedule

# Pod toleration to match
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "infra"
      effect: "NoSchedule"

# Remove taint
oc adm taint nodes worker-1 dedicated=infra:NoSchedule-

Example: Toleration Matching

Node has taint dedicated=gpu:NoSchedule. Only pods with matching toleration can schedule there.

Node Affinity

Copy# Hard requirement: only schedule in us-east zones
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values: [us-east-1a, us-east-1b]

Infra Node Setup Example

Common design workshop task: dedicate nodes for platform services (router, registry, monitoring) to avoid subscription costs and resource contention.

Copy# 1. Label workers as infra
oc label node worker-3 node-role.kubernetes.io/infra=""
oc label node worker-4 node-role.kubernetes.io/infra=""

# 2. Taint infra nodes (prevent user workloads)
oc adm taint nodes worker-3 worker-4 node-role.kubernetes.io/infra=reserved:NoSchedule
oc adm taint nodes worker-3 worker-4 node-role.kubernetes.io/infra=reserved:NoExecute

# 3. Move Ingress Controller to infra nodes
oc patch ingresscontroller/default -n openshift-ingress-operator --type=merge -p \
  '{"spec":{"nodePlacement":{"nodeSelector":{"matchLabels":{"node-role.kubernetes.io/infra":""}},"tolerations":[{"key":"node-role.kubernetes.io/infra","value":"reserved","effect":"NoSchedule"},{"key":"node-role.kubernetes.io/infra","value":"reserved","effect":"NoExecute"}]}}}'

# 4. Move Monitoring to infra nodes (edit ConfigMap)
oc -n openshift-monitoring edit cm cluster-monitoring-config
# Add nodeSelector + tolerations under prometheusK8s, alertmanagerMain

# 5. Move Registry to infra nodes
oc patch configs.imageregistry.operator.openshift.io/cluster --type=merge -p \
  '{"spec":{"nodeSelector":{"node-role.kubernetes.io/infra":""},"tolerations":[{"key":"node-role.kubernetes.io/infra","value":"reserved","effect":"NoSchedule"},{"key":"node-role.kubernetes.io/infra","value":"reserved","effect":"NoExecute"}]}}'

Core Resources

The fundamental Kubernetes and OpenShift objects that compose every application deployment.

Container Image Lifecycle

Before a container can run in a Pod, its image must be built and pushed to a registry. Here is how source code becomes a running container.

Copy# Example Dockerfile
FROM registry.access.redhat.com/ubi9/python-311:latest
COPY . /app
RUN  pip install -r /app/requirements.txt
EXPOSE 8080
CMD  ["python", "/app/main.py"]

# Build, tag, push
podman build -t my-app:v1.0 .
podman tag my-app:v1.0 registry.example.com/team/my-app:v1.0
podman push registry.example.com/team/my-app:v1.0

• Dockerfile / Containerfile — Text file with instructions to build image. Each instruction (FROM, COPY, RUN) creates a layer. Use multi-stage builds to reduce final image size.
• Build — podman build or buildah bud executes Dockerfile. OpenShift also supports BuildConfig for in-cluster builds (S2I, Docker, Custom strategies).
• Layers — Images are stacked read-only layers. Base OS at bottom, app code at top. Shared layers between images save disk and pull time.
• Tag — Names image with registry/repository:tag format. Use semantic versioning (:v1.2.0) not :latest in production.
• Push — Uploads layers to registry. OpenShift has built-in registry at image-registry.openshift-image-registry.svc:5000. External: Quay, Docker Hub, private registries.
• Pull — When pod starts, kubelet tells CRI-O to pull image. Pull policy: Always, IfNotPresent, Never. Use ImagePullSecret for private registries.

How Resources Wire Together

Each resource references others via specific YAML fields. Understanding these connections is key to building and debugging applications.

Resource Quick Reference

Pod Anatomy — How a Pod Uses Resources

A Pod consumes configuration, secrets, and storage by referencing other Kubernetes objects. Compare this to how a traditional VM gets its config.

ConfigMaps & Secrets

Both decouple configuration from the container image (shown wired into the Pod above). A ConfigMap holds non-sensitive key-value data; a Secret holds sensitive data and is RBAC-protected. A pod consumes either as environment variables or as files in a mounted volume.

Copy# ConfigMap — non-sensitive config
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: info
  app.properties: |
    timeout=30
    retries=3
---
# Secret — sensitive data (stringData is auto base64-encoded on apply)
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  username: appuser
  password: s3cr3t-p@ss

Copy# Consume inside a container — three patterns
env:
  - name: LOG_LEVEL          # single key as env var
    valueFrom:
      configMapKeyRef: { name: app-config, key: LOG_LEVEL }
envFrom:
  - secretRef: { name: db-credentials }   # every key as env vars
volumeMounts:
  - name: config-vol            # mount as files
    mountPath: /etc/app

Secret Types

Deployment In Depth

A Deployment is the primary controller for stateless workloads. It creates and manages ReplicaSets, which in turn create Pods. When you update a Deployment (e.g., new image), it creates a new ReplicaSet and gradually shifts pods from old to new — this is a rolling update.

Copy# Create a deployment
oc create deployment nginx --image=nginx:1.25 --replicas=3

# Update image (triggers rolling update)
oc set image deployment/nginx nginx=nginx:1.26

# Check rollout status
oc rollout status deployment/nginx

# Rollback to previous version
oc rollout undo deployment/nginx

# View revision history
oc rollout history deployment/nginx

Namespaces & Projects

Logical isolation boundaries — how OpenShift organizes and secures multi-tenant workloads.

Namespace vs Project

A Namespace is the Kubernetes-native scope for namespaced resources. An OpenShift Project is an alternative, user-facing representation of a Namespace that can carry OpenShift display annotations and is exposed through the Project API. The important distinction is the creation workflow, not two independent isolation objects.

Project Anatomy

Default System Namespaces

Governance Resources

Multi-Tenancy Model

OpenShift provides soft multi-tenancy via namespace isolation. Each team gets their own Project with:

• RBAC — Role-based access scoped to the project. Users cannot see other projects by default.
• Resource Quotas — Cluster admin sets CPU/memory/pod limits per project
• Network Isolation — NetworkPolicy restricts cross-namespace traffic
• Node Isolation — Taints + tolerations can dedicate nodes per team

Common Commands

# Create a new project (with default RBAC)
oc new-project my-app --display-name="My Application"

# Switch to a project
oc project my-app

# List all projects you have access to
oc get projects

# Admin: create project for another user
oc adm new-project team-backend --admin=alice

# Set resource quota
oc create quota compute --hard=cpu=8,memory=16Gi,pods=20 -n my-app

# View current resource usage vs quota
oc describe quota -n my-app

# Delete a project (WARNING: deletes all resources inside)
oc delete project my-app

Pod Lifecycle

Understanding how pods are created, scheduled, run, and terminated in OpenShift.

What is a Pod?

A pod is the smallest deployable unit in Kubernetes/OpenShift. It wraps one or more containers that share the same network namespace (IP address), storage volumes, and lifecycle. Most pods run a single application container.

Pod Lifecycle Flow

Resource Management

Copyresources:
  requests:
    cpu: "250m"     # 0.25 CPU cores guaranteed
    memory: "256Mi"  # 256 MiB guaranteed
  limits:
    cpu: "1000m"   # Max 1 CPU core (throttled beyond)
    memory: "512Mi"  # Max 512 MiB (OOMKilled beyond)

Quality of Service (QoS) Classes

The kubelet derives a QoS class for every pod from its requests and limits — you don’t set it directly. QoS decides eviction order when a node runs low on memory: lower-priority pods are killed first to protect critical ones.

Health Probes

OpenShift uses three probe types (shown in the lifecycle diagram above) to manage container health and traffic. Each can use an httpGet, tcpSocket, exec, or grpc handler.

Copycontainers:
  - name: web
    startupProbe:           # slow boot: allow up to 30 × 10s = 300s
      httpGet: { path: /healthz, port: 8080 }
      failureThreshold: 30
      periodSeconds: 10
    livenessProbe:          # restart if deadlocked
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
    readinessProbe:         # pull from load balancer until deps are up
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5

Deployments & Workloads

How to deploy applications and choose the right workload controller on OpenShift.

Deployment Methods

Deployment Sources

Deployment Methods

Copy# Deploy from container image
oc new-app --image=registry.example.com/team/my-app:v1.0 --name=my-app

# Deploy from git repo (S2I auto-detect)
oc new-app https://github.com/team/my-app.git --name=my-app

# Deploy from Helm chart
helm install my-app ./chart --set replicas=3

# Deploy from YAML
oc apply -f deployment.yaml

# Deploy with Kustomize overlays
oc apply -k overlays/production/

# Expose app externally
oc expose svc/my-app

Workload Controllers

Deployment Strategies

Autoscaling

Automatically adjust pod replicas, container resources, and cluster capacity to match workload demand.

Why Autoscaling?

Static resource allocation wastes capacity during low demand and causes outages during spikes. OpenShift provides four complementary autoscaling mechanisms that operate at two distinct levels:

• Pod-level scaling — adjusts the number of pod replicas (HPA) or the CPU/memory requests on each pod (VPA).
• Node-level scaling — adds or removes worker nodes when the cluster itself runs out of schedulable capacity (Cluster Autoscaler + MachineAutoscaler).

Autoscaling Architecture

Autoscaling Mechanisms Comparison

HPA — Horizontal Pod Autoscaler

HPA watches metrics (CPU utilisation, memory, or custom/external metrics) and adjusts the replicas field on a target workload controller. It runs a control loop every 15 seconds by default and uses the autoscaling/v2 API for multi-metric and custom metric support.

HPA YAML Example

CopyapiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
  namespace: my-project
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70    # scale up when avg CPU > 70%
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # wait 5 min before scaling down
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

Copy# Quick HPA via CLI
oc autoscale deployment/my-app --min=2 --max=10 --cpu-percent=70

# Check HPA status
oc get hpa my-app-hpa
# NAME          REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS
# my-app-hpa    Deployment/my-app  45%/70%   2         10        3

VPA — Vertical Pod Autoscaler

VPA analyses historical and real-time resource usage and recommends (or applies) right-sized CPU and memory requests. On OpenShift it is delivered by the VerticalPodAutoscaler Operator installed from OperatorHub.

VPA Modes

VPA YAML Example

CopyapiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
  namespace: my-project
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: Auto         # Off | Initial | Auto
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 64Mi
        maxAllowed:
          cpu: 2
          memory: 2Gi

Cluster Autoscaler

The Cluster Autoscaler is a cluster-scoped singleton CR that watches for unschedulable pods (pods stuck in Pending because no node has enough resources). When detected, it triggers a scale-up of the appropriate MachineSet. It also scales nodes down when they are underutilised and their pods can be rescheduled elsewhere.

ClusterAutoscaler YAML

CopyapiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default             # must be named "default" (singleton)
spec:
  podPriorityThreshold: -10  # ignore pods with priority below this
  resourceLimits:
    maxNodesTotal: 24        # hard cap on total cluster nodes
    cores:
      min: 8
      max: 256
    memory:
      min: 16
      max: 1024              # GiB
  scaleDown:
    enabled: true
    delayAfterAdd: 10m      # wait after adding a node
    delayAfterDelete: 5m
    unneededTime: 5m        # node must be underused this long
  balanceSimilarNodeGroups: true

MachineAutoscaler

A MachineAutoscaler sets the minReplicas and maxReplicas bounds for a single MachineSet. Create one MachineAutoscaler per MachineSet (typically one per availability zone or instance type). The Cluster Autoscaler uses these bounds when deciding which MachineSet to scale.

MachineAutoscaler YAML

CopyapiVersion: autoscaling.openshift.io/v1beta1
kind: MachineAutoscaler
metadata:
  name: worker-us-east-1a
  namespace: openshift-machine-api
spec:
  minReplicas: 1
  maxReplicas: 6
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: cluster-abc-worker-us-east-1a

How They Work Together

In a well-configured cluster, autoscaling forms a chain reaction:

1. HPA increases pod replicas when request load rises.
2. New pods go Pending if existing nodes lack capacity.
3. Cluster Autoscaler detects unschedulable pods and picks a MachineSet that can satisfy the request.
4. MachineAutoscaler bounds constrain how many new Machines (nodes) that MachineSet can add.
5. The MachineSet creates a new Machine → the Machine API provisions a VM/instance → it joins as a new worker node.
6. Pending pods are scheduled onto the new node.
7. When load drops, HPA reduces replicas, the Cluster Autoscaler detects underused nodes, and scales them down (respecting PDBs).

Installation & Bootstrap

How OpenShift clusters are installed across different platforms, and the bootstrap process that brings a cluster to life.

Installation by Platform

VMware vSphere — IPI vs UPI

VMware vSphere supports both IPI and UPI. IPI automates everything; UPI gives full control at cost of manual steps.

IPI vs UPI Decision Matrix

Bootstrap Process

The bootstrap process is a temporary, self-destructing sequence that brings the cluster from zero to a running control plane.

Network Connectivity Ports

Agent-Based Installer

Generates a single bootable ISO containing the Assisted Service, discovery agent, and OCP release image. Fully self-contained — no external service or internet needed. One host acts as the rendezvous host, running the Assisted Service locally.

Disconnected / Air-Gapped Installation

Bare Metal — Assisted Installer

Red Hat’s SaaS-hosted installation service at console.redhat.com or deployable on-prem via the Infrastructure Operator (part of MCE/RHACM). Provides a web UI and REST API for guided cluster deployment.

Networking & Routes

How traffic flows from external clients to your application pods inside OpenShift.

Network Architecture

OpenShift has three distinct networks, each serving a different purpose:

• Host Network (Physical) — Real routable IPs on node NICs. Carries API (:6443), etcd (:2379), kubelet (:10250), and Geneve tunnel traffic. The only network with actual packets on the wire.
• Service Network (Virtual) — ClusterIP range 172.30.0.0/16. No real interface exists — OVN-Kubernetes implements via OVS flow rules. CoreDNS resolves service names to ClusterIPs.
• Pod Network (Overlay) — Each pod gets a unique IP from 10.128.0.0/14. Each node gets a /23 subnet (510 IPs). Cross-node traffic tunneled via Geneve (UDP 6081) over Host Network.

Ingress Traffic Flow

How Services Work

A Service provides a stable VIP and DNS name for a set of pods. It uses label selectors to dynamically discover backend pods. When pods scale or restart, endpoints update automatically.

Routes & Ingress Controller

A Route exposes a Service to external traffic via a hostname. The Ingress Controller (HAProxy) watches Route objects and configures proxying rules automatically.

MetalLB — LoadBalancer Services on Bare Metal

On a cloud provider, a Service of type LoadBalancer automatically gets a real external IP via the cloud's LB API. Bare-metal clusters have no such API — MetalLB fills that gap, assigning IPs from a pool you own and announcing them to the local network so external routers/switches know where to send traffic.

Copy# Pool of external IPs MetalLB can hand out
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: apps-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.240-192.168.1.250

---
# Announce that pool via Layer 2 (ARP/NDP)
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: apps-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - apps-pool

Cluster Network Defaults

OVN-Kubernetes

OVN-Kubernetes is the default CNI plugin since OpenShift 4.12. It replaces the legacy OpenShift SDN.

• Encapsulation: Uses Geneve tunnels (not VXLAN)
• Logical networking: Creates logical switches, routers, ports, ACLs, and load balancers
• No kube-proxy: OVN handles service load balancing natively via OVS flows
• NetworkPolicy: Enforced via OVN ACLs for microsegmentation

EgressIP

By default, outbound traffic from pods uses the node IP as source. This is a problem when external firewalls need to whitelist your cluster — you'd need to allow ALL node IPs, and they change when nodes scale. EgressIP assigns a predictable, static source IP to all outbound traffic from a namespace.

Copy# 1. Label nodes as egress-assignable
oc label node worker-1 k8s.ovn.org/egress-assignable=""
oc label node worker-2 k8s.ovn.org/egress-assignable=""

# 2. Create EgressIP object
oc apply -f - <<EOF
apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: my-app-egress
spec:
  egressIPs:
    - 192.168.1.155
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-app
EOF

# 3. Verify EgressIP assignment
oc get egressip
oc get egressip my-app-egress -o yaml

Multus CNI

Multus is a meta-plugin — it does not provide networking itself but delegates to other CNI plugins. Every OpenShift pod already gets a primary interface (eth0) from OVN-Kubernetes. Multus lets you attach additional interfaces (e.g., net1, net2) using plugins like macvlan, SR-IOV, bridge, or IPVLAN. This is essential for telco/NFV workloads that need dedicated data-plane networks, storage networks, or direct L2 access to physical infrastructure.

Secondary CNI Plugin Comparison

CopyapiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
  namespace: my-project
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "macvlan",
    "master": "ens4",
    "mode": "bridge",
    "ipam": {
      "type": "whereabouts",
      "range": "192.168.1.0/24",
      "exclude": ["192.168.1.0/32", "192.168.1.1/32"]
    }
  }'

CopyapiVersion: v1
kind: Pod
metadata:
  name: my-multi-nic-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf   # attach net1
spec:
  containers:
  - name: app
    image: registry.redhat.io/ubi9/ubi:latest
    command: ["sleep", "infinity"]

# Verify inside the pod:
# oc exec my-multi-nic-pod -- ip a
#   eth0  → 10.128.2.15  (OVN-K, cluster network)
#   net1  → 192.168.1.50 (macvlan, physical network)

Copy# Step 1: SriovNetworkNodePolicy — configure VFs on physical NIC
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: policy-dpdk
  namespace: openshift-sriov-network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"
  numVfs: 8
  nicSelector:
    pfNames: ["ens5f0"]
  deviceType: vfio-pci          # for DPDK; use netdevice for kernel driver
  resourceName: dpdk_nic

---
# Step 2: SriovNetwork — auto-creates NetworkAttachmentDefinition
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: sriov-dpdk-net
  namespace: openshift-sriov-network-operator
spec:
  networkNamespace: my-project
  resourceName: dpdk_nic
  ipam: '{ "type": "whereabouts", "range": "10.10.10.0/24" }'

Deployment Strategies via Routes

NetworkPolicy

NetworkPolicy

Kubernetes-native microsegmentation — control which pods can talk to which at the network level.

How NetworkPolicy Works

Copy# Allow DNS egress (add to every namespace with deny-all)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: openshift-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

Copy# View policies in a namespace
oc get networkpolicy -n my-app

# Test connectivity between pods
oc exec deploy/frontend -- curl -s -o /dev/null -w "%{http_code}" http://backend:8080/health

Storage & CSI Drivers

How persistent storage works in OpenShift using the Container Storage Interface.

Storage Architecture

Static vs Dynamic Provisioning

Reclaim Policies

Volume Binding Mode

volumeBindingMode is a field on the StorageClass that controls when a PVC is actually bound to a PV — immediately on PVC creation, or only once a Pod that uses it has been scheduled to a node.

Access Modes

CSI Architecture Details

Complete Example — PVC + Deployment

Create a PersistentVolumeClaim, then reference it in a Deployment. StorageClass triggers dynamic provisioning automatically.

Copy# 1. PersistentVolumeClaim — request 10Gi block storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
  namespace: my-app
spec:
  accessModes:
    - ReadWriteOnce         # Single node read-write
  storageClassName: thin-csi  # StorageClass name (vSphere thin disk)
  resources:
    requests:
      storage: 10Gi         # Requested size

Copy# 2. Deployment — mount PVC into container
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: app
          image: registry.example.com/my-app:1.0
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: data
              mountPath: /var/data    # Path inside container
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: app-data    # References PVC above

Copy# Apply and verify
oc apply -f pvc.yaml -f deployment.yaml

# Check PVC is Bound
oc get pvc -n my-app
# NAME       STATUS   VOLUME           CAPACITY   ACCESS   STORAGECLASS
# app-data   Bound    pvc-a1b2c3d4..   10Gi       RWO      thin-csi

# Verify mount inside pod
oc exec deploy/my-app -- df -h /var/data

Security & SCC

How OpenShift enforces security at the pod level using Security Context Constraints.

Security Context Constraints (SCC)

SCC are an OpenShift-specific security mechanism that controls what a pod can do at the OS level: run as root, access host network, mount volumes, use capabilities, etc. They are evaluated during pod admission.

SCC Admission Flow

SCC Controls

Security Context Constraints control the following OS-level capabilities for pods:

RBAC — Role-Based Access Control

RBAC in OpenShift controls who can do what on which resources.

• Rules — Define verbs (get, list, create, update, delete) + resources (pods, services) + API groups
• Roles — A collection of rules. Role is namespace-scoped; ClusterRole is cluster-scoped
• Bindings — Bind subjects (users, groups, ServiceAccounts) to a Role. RoleBinding is namespace-scoped; ClusterRoleBinding is cluster-scoped

Cluster Role vs Local Role

OpenShift has two scopes for roles: ClusterRole (cluster-scoped, reusable across namespaces) and Role (namespace-scoped, applies only within its namespace). The binding type determines where permissions take effect:

RBAC Authorization Flow

Default Cluster Roles

ServiceAccounts

Every namespace has a default ServiceAccount. Pods use it for API authentication. Custom SAs isolate permissions per workload. Tokens are auto-mounted at /var/run/secrets/kubernetes.io/serviceaccount/token.

Copy# Create a ServiceAccount
oc create sa my-app -n my-namespace

# Grant a role to a user in a namespace
oc adm policy add-role-to-user edit developer -n my-project

# Grant cluster-admin to a user
oc adm policy add-cluster-role-to-user cluster-admin admin-user

# Check if a user can perform an action
oc auth can-i create deployments -n my-project --as developer

# List role bindings in a namespace
oc get rolebindings -n my-project

# Remove self-provisioner from all authenticated users
oc adm policy remove-cluster-role-from-group self-provisioner system:authenticated:oauth

OAuth Authentication Flow

cert-manager — Certificate Lifecycle

cert-manager automates issuing and renewing X.509 certificates — the day-1 pain it removes is manually generating certs, copying them into Secrets, and remembering to rotate them before they expire.

CopyapiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform-team@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: openshift-default

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-tls
  namespace: my-app
spec:
  secretName: myapp-tls-secret
  dnsNames:
    - myapp.apps.cluster.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Logging & Monitoring

Core platform monitoring plus optional Operators for log collection, storage, and console integration.

Monitoring Architecture

OpenShift ships a fully integrated monitoring stack managed by the Cluster Monitoring Operator. All components are deployed automatically and run in the openshift-monitoring namespace.

Monitoring Components

Logging Architecture

Logging is not installed by default with core OCP. A supported Loki-based deployment uses Loki for storage, Vector for collection, and the ClusterLogForwarder CR for routing. Layered Operators work together: Logging Operator (deploys Vector), Loki Operator (manages LokiStack), and Cluster Observability Operator (console integration).

Logging Components

Copy# Check monitoring stack health
oc get pods -n openshift-monitoring
oc get pods -n openshift-user-workload-monitoring

# Check logging stack health
oc get pods -n openshift-logging

# View Prometheus alerts firing
oc get prometheusrules -A
oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}'

# Check log collector status
oc get clusterlogforwarder -n openshift-logging
oc logs ds/collector -n openshift-logging --tail=20

Operators & OLM

How OpenShift uses the Operator pattern to manage platform components and applications.

What is an Operator?

An Operator encodes human operational knowledge into software. It watches Custom Resources (CRs) and takes automated action: provisioning, scaling, backup, upgrades, failover. OpenShift itself is managed by ~30 built-in Cluster Operators.

Operator Lifecycle

Operator Pattern — How It Works

An Operator is a controller that watches Custom Resources (CRDs) and runs a reconciliation loop: observe current state → compare with desired state → take corrective action. This encodes human operational knowledge into software.

Common Operators

Copy# List installed operators (ClusterServiceVersions)
oc get csv -A

# List subscriptions
oc get sub -A

# Check pending install plans (Manual approval)
oc get installplan -A

# Approve a pending install plan
oc patch installplan <name> -n <ns> --type merge -p '{"spec":{"approved":true}}'

# Check cluster operators status
oc get co

ACM — Advanced Cluster Management

Optional layered product for managing multiple OpenShift and Kubernetes clusters from a single hub.

What is ACM?

Red Hat ACM provides multi-cluster lifecycle management: provision clusters, enforce governance policies, deliver applications, aggregate observability, and search resources across the fleet.

Hub & Spoke Architecture

ACM Console — Cluster Lifecycle

Hosted Control Planes (HyperShift)

Through MCE, the ACM hub can run Hosted Control Planes: spoke clusters whose control planes (API server, etcd, controllers) run as pods on the hub, while only worker nodes live at the spoke site. This cuts per-cluster cost (no 3 dedicated control-plane machines), speeds up cluster creation to minutes, and decouples control-plane and worker upgrades — widely used for fleet, edge, and telco deployments.

Governance & Policy Framework

ACM’s Governance, Risk & Compliance (GRC) framework lets you define policies on the hub and distribute them to managed clusters. Each policy specifies what to check, where to apply it (via Placement), and how to respond (inform or enforce).

ACM Console — Governance Overview

Policy Types

Live Example — CertificatePolicy Catching an Expiring Certificate

A CertificatePolicy named check-cert monitors TLS secrets in the swongpai-helm namespace with a minimum duration of 300 hours. When a certificate falls below that threshold, ACM flags it as NonCompliant and shows exactly which secret is about to expire.

Step 1 — CertificatePolicy Configuration (minimum duration: 300h)

Step 2 — Violation Detected: apps-wildcard-tls Expiring

ACM Console — Policy Sets

How Policy Works

A Policy is a wrapper that contains one or more policy templates (ConfigurationPolicy, CertificatePolicy, etc.). The Policy is created on the hub, bound to target clusters via PlacementBinding + Placement, and distributed by the Policy Controller. On each managed cluster, a local controller evaluates the templates and reports compliance status back to the hub.

Policy Structure

Every Policy has three core settings:

ConfigurationPolicy — Compliance Types

ConfigurationPolicy is the most common and versatile template. It compares desired object definitions against actual cluster state using three compliance types:

ConfigurationPolicy Example

CopyapiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
  name: require-limitrange
spec:
  remediationAction: inform   # start with audit
  severity: medium
  namespaceSelector:
    include: ["*"]
    exclude: ["openshift-*", "kube-*"]
  object-templates:
    - complianceType: musthave
      objectDefinition:
        apiVersion: v1
        kind: LimitRange
        metadata:
          name: default-limits
        spec:
          limits:
            - type: Container
              default:
                cpu: "500m"
                memory: "512Mi"
              defaultRequest:
                cpu: "100m"
                memory: "128Mi"

CopyapiVersion: policy.open-cluster-management.io/v1
kind: ConfigurationPolicy
metadata:
  name: no-privileged-pods
spec:
  remediationAction: inform
  severity: high
  namespaceSelector:
    include: ["*"]
    exclude: ["openshift-*", "kube-*"]
  object-templates:
    - complianceType: mustnothave
      objectDefinition:
        apiVersion: v1
        kind: Pod
        metadata:
          namespace: "{{ .metadata.namespace }}"
        spec:
          containers:
            - securityContext:
                privileged: true

Full Policy Wrapper

A ConfigurationPolicy doesn’t work alone — it must be wrapped in a Policy and bound to clusters via PlacementBinding + Placement. One frequently missed prerequisite: Placement only selects clusters from ManagedClusterSets that are bound to the policy namespace via a ManagedClusterSetBinding. Here’s the complete set of resources:

Copy# 1. Policy — wraps ConfigurationPolicy template
apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: policy-require-limitrange
  namespace: open-cluster-management-policies
  annotations:
    policy.open-cluster-management.io/standards: NIST SP 800-53
    policy.open-cluster-management.io/categories: CM Configuration Management
    policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
spec:
  remediationAction: inform
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: require-limitrange
        spec:
          remediationAction: inform
          severity: medium
          namespaceSelector:
            include: ["*"]
            exclude: ["openshift-*", "kube-*"]
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: LimitRange
                metadata:
                  name: default-limits
                spec:
                  limits:
                    - type: Container
                      default:
                        cpu: "500m"
                        memory: "512Mi"
---
# 2. ManagedClusterSetBinding — Placement only selects clusters from
#    ManagedClusterSets bound to the policy namespace
apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
  name: global
  namespace: open-cluster-management-policies
spec:
  clusterSet: global
---
# 3. Placement — select target clusters
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: placement-require-limitrange
  namespace: open-cluster-management-policies
spec:
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchExpressions:
            - key: env
              operator: In
              values: ["production", "staging"]
---
# 4. PlacementBinding — links Policy ↔ Placement
apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
  name: binding-require-limitrange
  namespace: open-cluster-management-policies
placementRef:
  apiGroup: cluster.open-cluster-management.io
  kind: Placement
  name: placement-require-limitrange
subjects:
  - apiGroup: policy.open-cluster-management.io
    kind: Policy
    name: policy-require-limitrange

PolicyGenerator — GitOps for Policies

PolicyGenerator is a Kustomize plugin that eliminates boilerplate. Instead of writing Policy + PlacementBinding + Placement by hand for each rule, you write a compact policyGenerator.yaml and run kustomize build — it generates all three resources automatically. This is the recommended approach for managing policies in Git (GitOps).

How PolicyGenerator Works

PolicyGenerator Example

CopyapiVersion: policy.open-cluster-management.io/v1
kind: PolicyGenerator
metadata:
  name: gen-security-baseline
placementBindingDefaults:
  name: binding-security-baseline
policyDefaults:
  namespace: open-cluster-management-policies
  remediationAction: inform
  severity: medium
  consolidateManifests: false   # one ConfigurationPolicy per manifest file
  placement:
    labelSelector:
      matchExpressions:
        - key: env
          operator: In
          values: ["production", "staging"]
policies:
  - name: require-resource-limits
    manifests:
      - path: manifests/limitrange.yaml
    policyAnnotations:
      policy.open-cluster-management.io/standards: NIST SP 800-53
      policy.open-cluster-management.io/categories: CM Configuration Management
  - name: require-network-policies
    severity: high
    remediationAction: enforce   # override default for this policy
    manifests:
      - path: manifests/deny-all-netpol.yaml
      - path: manifests/allow-dns-netpol.yaml
    policyAnnotations:
      policy.open-cluster-management.io/standards: CIS Kubernetes Benchmark
  - name: require-pod-disruption-budget
    manifests:
      - path: manifests/pdb.yaml

Copy# kustomization.yaml — in the same directory as policyGenerator.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
  - policyGenerator.yaml
# Run: kustomize build --enable-alpha-plugins .
# Output: Policy + PlacementBinding + Placement for each entry

PolicyGenerator Workflow

Application Deployment

ACM delivers applications to managed clusters using ArgoCD ApplicationSets with ACM Placement. ArgoCD supports two deployment modes: Push (hub ArgoCD applies to remote clusters) and Pull (each cluster runs its own ArgoCD that pulls from Git). Both use the same ApplicationSet + Placement CRDs on the hub.

For ArgoCD fundamentals (Application, AppProject, sync/health status, drift detection) that this multi-cluster model builds on, see the GitOps & Pipelines section.

Push vs Pull — How to Choose

ApplicationSet with ACM Placement

Copy
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource:
        configMapRef: acm-placement
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: my-placement
        requeueAfterSeconds: 180
  template:
    metadata:
      name: 'my-app-{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/org/my-app.git
        targetRevision: main
        path: 'overlays/{{name}}'
      destination:
        server: '{{server}}'
        namespace: my-app

Live Example — ApplicationSet Deploying via GitOps

An ApplicationSet named timecheck pulls from a Git repo (github.com/Surote/simple-fastapi-deployment) via ArgoCD push model. ACM Placement targets the local cluster. The topology view shows the full resource tree: ApplicationSet → Placement → Cluster → Route + Deployment + Service → ReplicaSet → Pod.

ACM Console — ApplicationSet Overview (Push Model)

ACM Console — Application Topology View

OpenShift Console — Deployed Workload Running

Observability

ACM aggregates metrics from all managed clusters into a central Thanos deployment on the hub. Each managed cluster runs a metrics-collector (based on Prometheus) that pushes metrics to the hub’s Thanos Receive endpoint. Grafana dashboards on the hub provide fleet-wide visibility: cluster health, resource utilization, API server latency, etcd performance, and custom alerts across the entire fleet.

MultiClusterObservability CR

CopyapiVersion: observability.open-cluster-management.io/v1beta2
kind: MultiClusterObservability
metadata:
  name: observability
spec:
  observabilityAddonSpec:
    enableMetrics: true
    interval: 300   # collection interval (seconds)
  storageConfig:
    metricObjectStorage:
      name: thanos-object-storage   # Secret with S3 creds
      key: thanos.yaml
    statefulSetSize: 10Gi   # local cache per Thanos component
  retentionConfig:
    retentionResolutionRaw: 5d
    retentionResolution5m: 14d
    retentionResolution1h: 365d
  advanced:
    queryFrontend:
      replicas: 2
    receive:
      replicas: 3

Copy# thanos.yaml — referenced by MultiClusterObservability
type: s3
config:
  bucket: acm-observability
  endpoint: s3.openshift-storage.svc:443
  insecure: false
  access_key: <from ObjectBucketClaim Secret>
  secret_key: <from ObjectBucketClaim Secret>

What's New in Recent ACM Releases

ACS — Advanced Cluster Security

Optional layered product for full-lifecycle container security from build to runtime.

What is ACS?

Red Hat ACS (formerly StackRox) provides vulnerability management, compliance scanning, network segmentation, and runtime threat detection across your entire container environment. It secures workloads at every stage — build, deploy, and runtime — from a single centralized console.

Key Capabilities

ACS Architecture

ACS uses a hub-spoke model. Central runs on a dedicated cluster (or the ACM hub). Each protected cluster runs Secured Cluster components that report to Central via gRPC/mTLS.

Policy & Violation Flow

ACS ships with an extensive built-in security policy library covering CVEs, misconfigurations, runtime anomalies, and compliance standards. Policies evaluate at build, deploy, or runtime — violations trigger alerts, block deployments, or kill pods depending on enforcement configuration. Policies can also be managed declaratively as code via the SecurityPolicy CRD (GitOps-friendly).

Deploying ACS on OpenShift

ACS is deployed via the RHACS Operator from OperatorHub. Install Central on a hub cluster, then deploy SecuredCluster bundles on each managed cluster.

Deployment Steps

Central CR

CopyapiVersion: platform.stackrox.io/v1alpha1
kind: Central
metadata:
  name: stackrox-central-services
  namespace: stackrox
spec:
  central:
    exposure:
      route:
        enabled: true   # auto-create Route
    persistence:
      persistentVolumeClaim:
        claimName: stackrox-db
    db:
      isEnabled: Default   # managed PostgreSQL
      persistence:
        persistentVolumeClaim:
          claimName: central-db
  scanner:
    analyzer:
      scaling:
        autoScaling: Enabled
        maxReplicas: 5
        minReplicas: 2
        replicas: 3
    scannerComponent: AutoSense   # V4 when available

SecuredCluster CR

CopyapiVersion: platform.stackrox.io/v1alpha1
kind: SecuredCluster
metadata:
  name: stackrox-secured-cluster
  namespace: stackrox
spec:
  clusterName: production
  centralEndpoint: central-stackrox.apps.hub.example.com:443
  admissionControl: # 4.9+ console: single ON/OFF enforcement toggle
    listenOnCreates: true
    listenOnUpdates: true
    listenOnEvents: true   # runtime enforcement
    contactImageScanners: ScanIfMissing
  perNode:
    collector:
      collection: CORE_BPF   # only supported method since ACS 4.5
      imageFlavor: Regular
    taintToleration: TolerateTaints

Key Features Deep Dive

Vulnerability Management

ACS scans images using Scanner V4 — the default scanner since ACS 4.8 (the legacy StackRox Scanner is deprecated) — and correlates CVEs with deployment context: is the vulnerable package in the running container? Is the port exposed? Is there a known exploit? This produces a risk score that prioritizes remediation beyond raw CVSS. Since ACS 4.10, base-image layers are distinguished from application-added layers, so platform and app teams each see the CVEs they own.

ACS Console — Vulnerability Management

Network Segmentation

ACS builds a real-time network flow graph from Collector observations. Visualize actual pod-to-pod and pod-to-external traffic, then generate NetworkPolicy YAML from observed flows — zero-trust microsegmentation without manual policy writing.

• Visualize active network flows per namespace and deployment
• Detect unexpected connections (e.g., frontend directly reaching database)
• Auto-generate NetworkPolicy from baseline traffic patterns
• Simulate policy changes before applying

ACS Console — Network Graph

Runtime Detection

Collector uses CO-RE BPF probes (Compile Once – Run Everywhere eBPF) on each node to monitor all container processes, network connections, and file access in real-time — without modifying containers, requiring sidecars, or loading per-kernel drivers.

• Process monitoring — detect unexpected binaries (crypto miners, reverse shells, nmap)
• Network monitoring — detect unexpected outbound connections or lateral movement
• File access — detect reads/writes to sensitive paths (/etc/shadow, /proc)
• Response actions — alert only, scale to zero, or kill pod immediately

Live Example — Process Discovery in Action

Run a few commands inside a pod terminal, and Collector picks them up instantly. Below: curl and tracepath executed in a toolbox pod, then flagged by ACS under Risk → Process Discovery with full parent-process lineage and timestamps.

Step 1 — Commands Executed in Pod Terminal

Step 2 — ACS Process Discovery Flags the Activity

Step 3 — Event Timeline with Process Lineage

ACS Console — Violations

Compliance Scanning

ACS continuously evaluates clusters against industry compliance standards and generates audit-ready reports. Current ACS delivers this through Compliance v2: the Compliance Operator runs the actual node/platform scans on each secured cluster, while ACS schedules scans, aggregates fleet-wide results, and produces reports.

• Supported standards: CIS Kubernetes Benchmark, NIST SP 800-190, PCI DSS 4.0, HIPAA, NERC-CIP
• Per-cluster and per-namespace compliance dashboards
• Export PDF/CSV reports for auditors
• Requires the Compliance Operator installed on each secured cluster for node-level checks

ACS Console — CIS-OCP Compliance Coverage (98% passing)

ACS Console — Compliance Check Detail (NetworkPolicy requirement)

CI/CD Integration (roxctl)

The roxctl CLI integrates ACS into CI/CD pipelines to shift security left — catch issues before deployment.

• roxctl image check — evaluate image against all deploy-time policies
• roxctl image scan — scan image for CVEs and return results as JSON
• roxctl deployment check — validate deployment YAML against policies
• Exit code non-zero on policy violation — fails the pipeline

Policy as Code (SecurityPolicy CRD)

ACS policies can be managed declaratively with the SecurityPolicy custom resource on the Central cluster — store policies in Git, apply via ArgoCD, and keep security rules under the same GitOps review flow as application manifests.

CopyapiVersion: config.stackrox.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: no-latest-tag
  namespace: stackrox   # must be on the Central cluster
spec:
  policyName: "Image uses :latest tag"
  description: "Forbid mutable :latest tags in production"
  severity: HIGH_SEVERITY
  categories:
    - DevOps Best Practices
  lifecycleStages:
    - BUILD
    - DEPLOY
  scope:
    - cluster: production
  policySections:
    - sectionName: Image tag
      policyGroups:
        - fieldName: Image Tag
          values:
            - value: latest

Live Example — "Image uses :latest tag" Policy as Code

A SecurityPolicy CRD deployed via GitOps creates the "Image uses :latest tag" policy in ACS. The policy is marked Externally managed (origin), meaning it’s controlled by the CRD — not editable in the UI. When a deployment uses :latest, ACS raises a deploy-time violation.

Step 1 — Policy in ACS Policy Management (Externally Managed)

Step 2 — Policy Details, Behavior & Criteria

Step 3 — Violation Triggered on Deployment

What's New in Recent ACS Releases

ODF — OpenShift Data Foundation

Optional layered storage product powered by Ceph, providing block, file, and object services.

What is ODF?

OpenShift Data Foundation (ODF) is Red Hat's software-defined storage solution for OpenShift, built on Ceph and managed by the Rook operator. It provides block (RBD), file (CephFS), and S3-compatible object (NooBaa MCG) storage as native Kubernetes resources. ODF eliminates external storage array dependencies by running storage services directly on OpenShift nodes with local or attached disks.

Key Capabilities

ODF Architecture

Data Flow: PVC to Disk

When a pod requests persistent storage, the request flows through Kubernetes and ODF layers. The path differs by storage type: RBD for block (RWO) and CephFS for shared file (RWX). Both ultimately write to OSDs on physical disks, distributed by the CRUSH algorithm.

Storage Types Comparison

ODF exposes three distinct storage types. Choose based on access mode requirements and workload characteristics.

Deploying ODF on OpenShift

ODF is deployed via the ODF Operator from OperatorHub. The operator installs the Rook-Ceph operator, NooBaa operator, and CSI drivers. Internal mode uses local disks on labeled nodes; external mode connects to a pre-existing Ceph cluster. A third option, provider mode, lets one dedicated ODF cluster serve storage to multiple consumer OpenShift clusters.

Deployment Steps

StorageCluster CR

CopyapiVersion: ocs.openshift.io/v1
kind: StorageCluster
metadata:
  name: ocs-storagecluster
  namespace: openshift-storage
spec:
  manageNodes: false
  monDataDirHostPath: /var/lib/rook
  storageDeviceSets:
    - name: ocs-deviceset
      count: 1   # devices per node
      dataPVCTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 512Gi   # size per OSD
          storageClassName: localblock
          volumeMode: Block
      placement: {}
      portable: true
      replica: 3   # 3-way replication
  multiCloudGateway:
    reconcileStrategy: standalone

Example PVCs

Copy# RBD block volume (RWO) — databases, general storage
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
  namespace: my-app
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: ocs-storagecluster-ceph-rbd
---
# CephFS shared volume (RWX) — shared across pods/nodes
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-uploads
  namespace: my-app
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: ocs-storagecluster-cephfs

OCP Console — PVCs Bound to ODF StorageClasses (RWO ceph-rbd + RWX cephfs)

Key Concepts Deep Dive

CRUSH Algorithm & Failure Domains

CRUSH (Controlled Replication Under Scalable Hashing) is Ceph's data placement algorithm. Instead of centralized lookup tables, CRUSH computes where data should be stored algorithmically based on the cluster topology. This enables Ceph to scale without metadata bottlenecks.

• Failure domains — CRUSH distributes replicas across fault boundaries: host (default), rack, zone, or region. No two replicas of the same object land in the same failure domain.
• CRUSH map — Defines the storage hierarchy: root → datacenter → rack → host → OSD. ODF auto-generates this from node topology labels.
• CRUSH rules — Bind pools to specific device classes (SSD vs HDD) and failure domain levels. ODF creates default rules for replicated and erasure-coded pools.
• Rebalancing — When nodes/OSDs are added or removed, CRUSH recomputes placement. Only the minimum necessary data migrates (no full reshuffle).

Replication & Erasure Coding

ODF supports two data protection strategies. Replication (default) writes N full copies across failure domains. Erasure coding splits data into data + parity chunks for better space efficiency at higher CPU cost.

• 3-way replication (default) — Every object stored 3 times across 3 failure domains. Data survives loss of 2 replicas; cluster tolerates loss of 1 failure domain while remaining fully available. ~33% storage efficiency. Best for performance-sensitive workloads.
• Erasure coding (EC) — Splits data into k data chunks + m parity chunks (e.g., 4+2). Tolerates loss of m chunks. ~67% efficiency (4+2). Higher CPU overhead; best for large, cold, or archival data.
• Pool-level setting — Each Ceph pool is either replicated or erasure-coded. RBD pools are typically replicated (latency-sensitive). CephFS data pools can use EC for bulk storage.
• Minimum cluster size — 3 failure domains (hosts) for replication, k+m failure domains for EC. ODF internal mode requires minimum 3 nodes.

Snapshots & Clones

ODF supports CSI VolumeSnapshots and clones for both RBD and CephFS volumes. Snapshots are point-in-time, copy-on-write captures. Clones create instant read-write copies from snapshots.

• VolumeSnapshot — Create via VolumeSnapshot CR referencing a PVC. Uses VolumeSnapshotClass (one per storage type). Copy-on-write means only changed blocks consume additional space.
• Clone from snapshot — Create a new PVC with dataSource referencing a VolumeSnapshot. Instant provisioning; new PVC is a full read-write copy.
• PVC-to-PVC clone — Direct clone without intermediate snapshot. The CSI driver handles it transparently.
• Backup integration — OADP (OpenShift API for Data Protection) uses CSI snapshots for Velero-based backup/restore workflows.

NooBaa Multi-Cloud Gateway (MCG)

NooBaa provides an S3-compatible object storage API on OpenShift. It aggregates multiple backing stores behind a single S3 endpoint, with data placement policies for tiering, mirroring, and spreading.

• BackingStore — CRD defining a storage target: ODF (Ceph RGW), AWS S3 bucket, Azure Blob container, GCP bucket, or PV-based store. Multiple BackingStores can coexist.
• BucketClass — Defines data placement policy: spread (stripe across stores), mirror (replicate to all stores), or tier (hot/cold placement). Applied per-bucket.
• ObjectBucketClaim (OBC) — Kubernetes-native way to provision S3 buckets. Creates a ConfigMap (endpoint, bucket name) and Secret (access key, secret key) in the claiming namespace.
• Use cases — OADP backup target, ACM observability (Thanos), Loki log storage, ML datasets, and application object storage without external S3 dependency.

OCP Console — Object Bucket Claims (Loki OBC bound to NooBaa)

OCP Console — Bucket Contents (object upload & browsing)

OCP Console — Presigned URL Sharing (time-limited S3 access)

Disaster Recovery: Metro-DR vs Regional-DR

ODF supports two DR topologies, both orchestrated with ACM and the Ramen (OpenShift DR) operators. Metro-DR stretches a single Ceph cluster synchronously across two sites (RPO zero, distance-limited). Regional-DR asynchronously replicates between two independent ODF clusters (minutes of RPO, unlimited distance). Since ODF 4.20, DR-protected VMs can fail over individually rather than only as a whole namespace.

What's New in Recent ODF Releases

Quay — Enterprise Container Registry

Optional enterprise registry product for securing, scanning, and distributing container images at scale.

What is Red Hat Quay?

Red Hat Quay is an enterprise container image registry that provides secure storage, vulnerability scanning, and global distribution of container images. It supports OCI and Docker image formats, integrates deeply with OpenShift, and offers governance features for multi-team environments.

Key Features

Quay Architecture

Quay consists of multiple microservice components deployed as pods on OpenShift, managed by the Quay Operator via a single QuayRegistry custom resource.

Image Flow — Push to Pull

From developer workstation to running pod, every image passes through Quay’s security pipeline.

Deploying Quay on OpenShift

Quay is deployed via the Quay Operator from OperatorHub. Install the operator, then create a QuayRegistry custom resource. The operator manages all components (Clair, PostgreSQL, Redis, object storage) automatically.

Deployment Steps

MCG Integration — Object Storage for Quay

Quay stores image layers and blobs in S3-compatible object storage. When running on OpenShift with ODF, Multicloud Object Gateway (MCG) powered by NooBaa provides this storage natively — no external S3 service needed.

OCP Console — Quay Image Blobs in the MCG Bucket

QuayRegistry CR

CopyapiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: central
  namespace: quay-enterprise
spec:
  configBundleSecret: quay-config-bundle
  components:
    - kind: clair
      managed: true
    - kind: clairpostgres
      managed: true
    - kind: postgres
      managed: true
    - kind: redis
      managed: true
    - kind: objectstorage
      managed: true   # uses ODF/MCG via ObjectBucketClaim
    - kind: route
      managed: true
    - kind: horizontalpodautoscaler
      managed: true
    - kind: mirror
      managed: true
    - kind: monitoring
      managed: true
    - kind: tls
      managed: true

ObjectBucketClaim for MCG

CopyapiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: quay-bucket
  namespace: quay-enterprise
spec:
  generateBucketName: quay-bucket
  storageClassName: openshift-storage.noobaa.io
  additionalConfig:
    bucketclass: noobaa-default-bucket-class
---
# After OBC is bound, a Secret and ConfigMap are created:
#   Secret:    quay-bucket  (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
#   ConfigMap: quay-bucket  (BUCKET_NAME, BUCKET_HOST, BUCKET_PORT)

Key Features Deep Dive

Clair Security Scanning

Clair V4 (indexer/matcher architecture) automatically scans every image pushed to Quay against continuously updated vulnerability databases. Scans happen asynchronously — results appear in the Quay UI under each image tag. Supports indexing OCI and Docker v2 manifest formats.

Robot Accounts

Robot accounts are service-level credentials scoped to specific repositories or organizations. They replace shared human credentials in CI/CD pipelines and automated deployments.

• Organization robots — access all repos in the org (admin sets per-repo permissions)
• Repository robots — scoped to a single repo, least-privilege
• Generate Kubernetes pull secrets directly from Quay UI for use as imagePullSecrets

Repository Mirroring

Mirror external registries into Quay on a schedule. Essential for disconnected / air-gapped environments common in telco and government deployments.

• Mirror from Docker Hub, Red Hat registries, or any OCI registry
• Filter by tag pattern (regex) to mirror only needed tags
• Scheduled sync (hourly, daily, weekly)
• Mirrored repos are read-only in Quay — source of truth is upstream

Live Example — Mirroring from quay.io to Internal Quay

Mirror an upstream image (quay.io/rh_ee_swongpai/fast-localtime-check) into an internal Quay instance as repository swongpai/timesync. Set up a Robot Account for pull credentials, configure the mirror with a tag filter and sync schedule, trigger a sync, then watch the mirrored tag appear with a Clair scan.

Step 1 — Robot Account for Mirror Pull Credentials

Step 2 — Mirror Configuration on Internal Quay

Step 3 — Sync Logs Show Successful Mirror

Step 4 — Mirrored Tag Appears in Internal Quay

Geo-Replication

Replicate image data across multiple storage backends in different geographic regions. All Quay instances share a single metadata database but store blobs locally for low-latency pulls.

• Single global namespace — quay.example.com/org/repo:tag works from any region
• Each region stores its own copy of image layers
• Pulls served from nearest storage backend
• Requires shared PostgreSQL and shared Redis across all regions (or external managed equivalents)

Quota Management & Auto-Pruning

Control storage consumption per organization and repository to prevent unbounded growth.

• Set warning threshold and reject threshold per org
• Pushes rejected when quota exceeded
• Track consumption via Quay UI or API
• Auto-pruning policies — org- or repo-level rules that delete tags automatically by tag count (keep newest N) or tag age (delete older than X days); multiple policies with regex tag-pattern filters can coexist
• Combine with tag expiration (per-tag TTL) for full lifecycle control

What's New in Recent Quay Releases

Day-2 Operations

Upgrading, monitoring, and maintaining your OpenShift cluster over its lifetime.

Cluster Upgrade Flow

Control Plane Only Updates (formerly EUS-to-EUS)

Extended Update Support (EUS) releases are selected even-numbered minor versions with an extended support phase. A Control Plane Only update minimizes compute-node reboots between consecutive EUS releases by pausing all non-control-plane MachineConfigPools while the cluster completes two explicit minor-version updates: 4.y → 4.y+1 → 4.y+2.

The odd-numbered release is not bypassed: the cluster must reach it successfully before starting the target EUS update. Paused compute nodes remain on their earlier machine configuration until the target control plane is healthy and compatible layered Operators have been reviewed. Unpausing pools then applies the accumulated configuration, often with one compute-node reboot.

Update Channels

Control Plane Only Update Outline: 4.20 → 4.22

Copy# 0. Confirm 4.20 is healthy, fully updated, and offers the expected graph edges
oc get clusterversion,clusteroperators
oc get mcp
oc adm upgrade

# 1. Select the target EUS channel
oc adm upgrade channel eus-4.22

# 2. Pause EVERY non-control-plane MCP (worker and any custom pools)
oc patch mcp/worker --type merge -p '{"spec":{"paused":true}}'
# Repeat for custom pools, for example: oc patch mcp/infra ...

# 3. Update explicitly to a RECOMMENDED 4.21.z shown by "oc adm upgrade"
oc adm upgrade --to=<recommended-4.21.z>
# Wait until ClusterVersion and ClusterOperators are healthy at 4.21.z.
# Review/update installed OLM Operators before the next minor update.

# 4. Update explicitly to a RECOMMENDED 4.22.z shown by "oc adm upgrade"
oc adm upgrade --to=<recommended-4.22.z>
oc get clusterversion,clusteroperators

# 5. When the 4.22 control plane is healthy, unpause pools one at a time
oc patch mcp/worker --type merge -p '{"spec":{"paused":false}}'
oc get mcp --watch