Blog

If you need Helm v4 from Red Hat, you can grab the package manager for Kubernetes at https://mirror.openshift.com/pub/cgw/helm/4.1.4/

As OpenShift Container Platform (OCP) adoption continues to grow on IBM Power systems, customers are increasingly looking to combine advanced networking capabilities with on-premises PowerVM deployments. One area that often generates questions is how EgressIP behaves with a primary UserDefinedNetwork (UDN) and how traffic is routed from workloads attached to that network.

This walkthrough demonstrates how to deploy a Layer 2 primary UserDefinedNetwork, assign an EgressIP, and validate that outbound traffic from a pod traverses the UDN interface rather than the cluster default network. The example is particularly relevant for OpenShift on IBM Power, including PowerVM environments managed through PowerVC.

Why UserDefinedNetworks Matter

UserDefinedNetworks (UDNs) allow OpenShift administrators to create custom pod networking domains separate from the cluster’s default network. When deployed as a Primary network, the UDN becomes the pod’s default network interface and routing domain.

This can be useful for:

• Application isolation
• Dedicated routing paths
• Network segmentation
• Multi-tenant environments
• Advanced EgressIP scenarios

For organizations running OCP on IBM Power, UDNs provide the same flexibility available on other supported OpenShift platforms while leveraging the scale and resiliency of Power infrastructure.

Prerequisites

Before creating the EgressIP resource, choose a valid IP address that:

• Is currently unused
• Is routable from your worker nodes
• Exists on the same Layer 2 segment as the node interface hosting the EgressIP

⚠️ Replace EGRESS_IP_ADDRESS with a valid address from your environment.

For on-premises PowerVM deployments, this IP should belong to the same network segment as the worker nodes that will advertise the EgressIP.

Create the UDN-Enabled Namespace

Namespaces that use a primary UDN must be labeled appropriately.

cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: egressip-udn-repro
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
    egressip-test: "true"
EOF

The label k8s.ovn.org/primary-user-defined-network: “” signals that workloads in the namespace should attach to a primary UserDefinedNetwork.

The additional label egressip-test: “true” is later used by the EgressIP namespaceSelector.

Create a Layer 2 Primary UserDefinedNetwork

Create a primary Layer 2 UDN:

cat <<'EOF' | oc apply -f -
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: repro-udn-l2
  namespace: egressip-udn-repro
spec:
  topology: Layer2
  layer2:
    role: Primary
    subnets:
      - "10.100.0.0/24"
EOF

Wait for the network to become ready:

oc wait userdefinednetwork repro-udn-l2 -n egressip-udn-repro \
  --for condition=NetworkReady=True --timeout=60s

💡 Ensure the UDN subnet does not overlap with the cluster pod network, service network, or OVN join subnets. In this example, 10.100.0.0/24 is used as the UDN subnet.

Enable EgressIP Assignment on a Worker Node

Label a worker node so OVN-Kubernetes can host EgressIP assignments:

oc label node worker-0 k8s.ovn.org/egress-assignable=””

You can verify the label using:

oc get nodes –show-labels

Create the EgressIP Resource

cat <<'EOF' | oc apply -f -
apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: repro-egressip
spec:
  egressIPs:
    - "EGRESS_IP_ADDRESS"
  namespaceSelector:
    matchLabels:
      egressip-test: "true"
EOF

In a lab environment, an unused address was selected from the worker node network and associated with the node through an additional interface configured in PowerVC – and EGRESS_IP_ADDRESS was updated to match the unused address that was routable on the network.

Verify EgressIP Assignment

Confirm that OVN successfully assigned the EgressIP:

oc get egressip repro-egressip -o yaml

Review the status section and verify the address is assigned to the intended worker node.

Example:

status:        items:        – egressIP: 192.168.1.50          node: worker-0

Deploy a Test Pod

Create a simple UBI-based pod:

cat <<'EOF' | oc apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: egress-test-pod
  namespace: egressip-udn-repro
  labels:
    app: egress-test
spec:
  containers:
    - name: curl
      image: registry.access.redhat.com/ubi9/ubi-minimal:latest
      command: ["sleep", "infinity"]
      securityContext:
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        capabilities:
          drop: ["ALL"]
        seccompProfile:
          type: RuntimeDefault
EOF

Wait until the pod becomes ready:

oc wait pod egress-test-pod -n egressip-udn-repro --for condition=Ready --timeout=120s

Install curl

Access the pod:

oc rsh -n egressip-udn-repro egress-test-pod

Install curl:

microdnf –setopt=cachedir=/tmp/dnf-cache –setopt=keepcache=0 install -y curl

Validate External Connectivity

Generate outbound traffic:

curl -sv https://google.com/ –connect-timeout 10

You can replace the endpoint with any reachable external service that allows source IP verification.

Inspect Routing Inside the Pod

Review the routing table:

cat /proc/net/route

Example output:

Iface      Destination Gateway      ovn-udn1   00000000    0100640A      ovn-udn1   0000640A    00000000      eth0       0000800A    0102800A

Decoded, the routes appear as:

Destination	Gateway	Interface
0.0.0.0	10.100.0.1	ovn-udn1
10.100.0.0/24	Direct	ovn-udn1
10.128.0.0/22	10.128.2.1	eth0
100.64.0.0/16	10.128.2.1	eth0
100.65.0.0/16	10.100.0.1	ovn-udn1
172.30.0.0/16	10.100.0.1	ovn-udn1

The key observation is that the default route points to ovn-udn1, confirming that the primary UDN is acting as the pod’s primary network.

Verify Traffic Flow

Examine interface statistics:

cat /proc/net/dev

Example:

Inter-| Receive | Transmit      …      eth0:      446      5      ovn-udn1: 21828173 4698

The significantly higher packet and byte counts on ovn-udn1 indicate that application traffic is flowing through the UserDefinedNetwork rather than the cluster default interface.

This validates that:

1. The pod is attached to the primary UDN.
2. The default route is installed through ovn-udn1.
3. Outbound traffic follows the UDN path.
4. EgressIP can be applied to workloads selected through the namespace selector.

Conclusion

OpenShift’s UserDefinedNetwork capability provides a powerful way to create dedicated networking domains for applications while still taking advantage of platform services such as EgressIP. On IBM Power deployments, including PowerVM-based environments, this enables architects to combine network isolation with consistent outbound IP presentation.

In this validation, a primary Layer 2 UDN successfully became the pod’s default network, routes were installed through ovn-udn1, and traffic counters confirmed that outbound connections traversed the UDN. Combined with EgressIP assignment, this offers a flexible pattern for workloads that require network segmentation and predictable egress behavior on OpenShift running on IBM Power.

Ashwin Hendre’s latest guide walks through the requirements, networking considerations, IBM Cloud configuration, DNS setup, credential management, and installation workflow needed to build a private OpenShift Installer Provisioned Infrastructure on IBM PowerVS while maintaining outbound internet connectivity.

• ✅ Private cluster architecture
• ✅ IBM Cloud VPC and DNS configuration
• ✅ PowerVS IPI deployment prerequisites
• ✅ OpenShift 4.22 installation guidance
• ✅ Cloud Credential Operator (CCO) setup

see https://community.ibm.com/community/user/blogs/ashwin-hendre/2026/08/06/private-ocp-powervs-ipi

Based on Natalia Jordan’s IBM Community article covering IPI PowerVS bootstrap node access, VPC networking concepts, jump host setup, SSH key configuration, and bootstrap log collection for OpenShift IPI troubleshooting

You can view the article at https://community.ibm.com/community/user/blogs/natalia-jordan/2026/02/11/accessing-bootstrap-logs-through-vpc-ssh

IBM Power Systems Virtual Server (PowerVS) is a great fit for data-intensive workloads. In this post, we’ll walk through deploying a single-node Apache Cassandra 5.0 instance on PowerVS using a RHEL 10 Bring-Your-Own-License (BYOL) image, fully automated with Terraform and a cloud-init shell script.

By the end you’ll have:

• A running PowerVS LPAR (s1022) on RHEL 10
• IBM Semeru Runtime (OpenJ9) for Java on ppc64le
• Apache Cassandra 5.0 installed and ready to start
• A public network interface for SSH access

---

Architecture Overview

The deployment is split into two layers:

Infrastructure (Terraform)

Resource	Purpose
ibm_pi_network	Creates a public pub-vlan network for external access
ibm_pi_instance	Provisions the LPAR using the RHEL 10 BYOL stock image
null_resource	SSHes into the instance after boot and runs the init script

Configuration (init.sh)

Once the LPAR is up, Terraform copies a templated init.sh script to the instance and executes it over SSH. The script:

1. Registers the OS with Red Hat Subscription Manager (RHSM)
2. Enables the RHEL 10 ppc64le base and appstream repos
3. Updates the OS and installs tooling (wget, curl, jq, etc.)
4. Downloads and installs IBM Semeru Java 17 LTS for ppc64le from GitHub Releases
5. Registers the cassandra YUM repo and installs Apache Cassandra 5.0
6. Drops a cassandra-manager.service systemd unit (installed but not auto-started — you control when it runs)

---

Prerequisites

Before you begin, make sure you have:

• An IBM Cloud account with PowerVS permissions
• A pre-provisioned PowerVS workspace (get the workspace GUID from the UI)
• A PowerVS SSH key registered in the workspace
• Red Hat credentials (RHSM username and password) for a RHEL subscription
• Terraform ≥ 1.3 installed locally
• The IBM Cloud CLI with the PowerVS plugin (ibmcloud plugin install power-iaas)

RHEL 10 BYOL image: This example uses the stock catalog image ID 585ca713-f303-45f0-a836-e9dd4f4c8f3b (RHEL10-BYOL) which is already available in PowerVS — no COS bucket or manual image upload required.

---

Repository Layout

.
├── main.tf                  # Provider, network, instance, and provisioner
├── variables.tf             # Input variable declarations
├── outputs.tf               # Instance ID and IP outputs
├── init.sh                  # cloud-init / SSH-executed bootstrap script
└── terraform.tfvars.example # Template — copy to terraform.tfvars

---

Step 1 — Configure Your Variables

Copy the example variables file and fill in your values:

cp terraform.tfvars.example terraform.tfvars

Then edit terraform.tfvars:

ibmcloud_api_key      = "YOUR_IBM_CLOUD_API_KEY"
workspace_id          = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
region                = "us-south"
zone                  = "dal12"
ssh_key_name          = "my-powervs-ssh-key"
ssh_private_key_path  = "~/.ssh/id_rsa"
memory                = 16
processors            = 2
rhsm_username         = "your-redhat-username"
rhsm_password         = "your-redhat-password"

Never commit terraform.tfvars to source control — it contains your IBM Cloud API key and RHSM credentials. The .gitignore in this repo already excludes it, and all three sensitive variables are declared with sensitive = true in variables.tf so Terraform redacts them from plan/apply output.

Variable Reference

Variable	Description	Default
ibmcloud_api_key	IBM Cloud API key	—
workspace_id	PowerVS workspace GUID	—
region	IBM Cloud region (e.g. us-south)	—
zone	IBM Cloud zone (e.g. dal12)	—
ssh_key_name	SSH key registered in the PowerVS workspace	—
ssh_private_key_path	Local path to the matching private key	~/.ssh/id_rsa
memory	Instance memory in GiB	16
processors	Number of virtual processors	2
rhsm_username	Red Hat Subscription Manager username	—
rhsm_password	Red Hat Subscription Manager password	—

---

Step 2 — Understand the Terraform Configuration

Provider

main.tf pins the IBM Cloud provider at ~> 2.4.0 and the HashiCorp null provider for the remote-exec provisioner:

terraform {
  required_providers {
    ibm = {
      source  = "IBM-Cloud/ibm"
      version = "~> 2.4.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.0"
    }
  }
}

provider "ibm" {
  ibmcloud_api_key = var.ibmcloud_api_key
  region           = var.region
  zone             = var.zone
}

Public Network

A pub-vlan network is created to give the instance an external IP for SSH access:

resource "ibm_pi_network" "public_net" {
  pi_cloud_instance_id = var.workspace_id
  pi_network_name      = "cassandra-public-net"
  pi_network_type      = "pub-vlan"
}

PowerVS Instance

The LPAR is an s1022 system (Power10) using a shared processor, the stock RHEL 10 BYOL image, and attached to the public network:

resource "ibm_pi_instance" "cassandra_node" {
  pi_cloud_instance_id = var.workspace_id
  pi_memory            = var.memory
  pi_processors        = var.processors
  pi_instance_name     = "cassandra-rhel10"
  pi_proc_type         = "shared"
  pi_sys_type          = "s1022"
  pi_image_id          = "585ca713-f303-45f0-a836-e9dd4f4c8f3b"
  pi_key_pair_name     = var.ssh_key_name
  pi_health_status     = "WARNING"

  pi_network {
    network_id = ibm_pi_network.public_net.network_id
  }

  depends_on = [ibm_pi_network.public_net]
}

pi_health_status = "WARNING" lets Terraform proceed even while the instance is still booting, which is normal — the null_resource provisioner handles the wait via SSH.

Remote Provisioner

After the instance is up, Terraform templates init.sh with your RHSM credentials and copies it over SSH, then executes it:

resource "null_resource" "cassandra_init" {
  triggers = {
    instance_id = ibm_pi_instance.cassandra_node.instance_id
    script_hash = filemd5("${path.module}/init.sh")
  }

  connection {
    type    = "ssh"
    user    = "root"
    host    = ibm_pi_instance.cassandra_node.pi_network[0].external_ip
    agent   = true
    timeout = "10m"
  }

  provisioner "file" {
    content = templatefile("${path.module}/init.sh", {
      rhsm_username = var.rhsm_username
      rhsm_password = var.rhsm_password
    })
    destination = "/root/init.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "chmod +x /root/init.sh",
      "bash /root/init.sh",
    ]
  }
}

The script_hash trigger means Terraform will re-run the provisioner if you change init.sh, which is handy during development.

---

Step 3 — The Bootstrap Script (init.sh)

init.sh is a Bash script executed on the instance as root. Here’s what it does, step by step.

RHSM Registration

subscription-manager register \
  --username="${rhsm_username}" \
  --password="${rhsm_password}" \
  --force

subscription-manager repos \
  --enable=rhel-10-for-ppc64le-baseos-rpms \
  --enable=rhel-10-for-ppc64le-appstream-rpms

The --force flag handles re-registration gracefully if the instance was previously registered. The correct repo slugs for RHEL 10 on Power LE (ppc64le) are rhel-10-for-ppc64le-*.

OS Update and Tooling

dnf update -y
dnf install -y wget tar gzip jq curl

IBM Semeru Java 17 (ppc64le)

Cassandra requires Java, and IBM Semeru Runtime (OpenJ9) is the recommended JVM for Power. The script fetches the latest release URL dynamically from the GitHub API:

SEMERU_LATEST_URL=$(curl -s https://api.github.com/repos/ibmruntimes/semeru17-binaries/releases/latest \
  | jq -r '.assets[] | select(.name | contains("jdk_ppc64le_linux")) | select(.name | endswith(".tar.gz")) | .browser_download_url')

wget -qO /tmp/semeru.tar.gz "$SEMERU_LATEST_URL"
mkdir -p /opt/ibm/semeru
tar -xzf /tmp/semeru.tar.gz -C /opt/ibm/semeru --strip-components=1
rm -f /tmp/semeru.tar.gz

alternatives --install /usr/bin/java java /opt/ibm/semeru/bin/java 1
echo "export JAVA_HOME=/opt/ibm/semeru" > /etc/profile.d/semeru.sh
export JAVA_HOME=/opt/ibm/semeru
export PATH="$JAVA_HOME/bin:$PATH"

Installing under /opt/ibm/semeru keeps it clean and separate from the system Java. The alternatives registration makes java available system-wide.

Apache Cassandra 5.0

The official Apache Cassandra RPM repo is added and the package installed. --skip-broken is used because Cassandra’s bundled JVM dependency is skipped in favour of Semeru:

rpm --import https://downloads.apache.org/cassandra/KEYS

cat <<'EOF' > /etc/yum.repos.d/cassandra.repo
[cassandra]
name=Apache Cassandra 
baseurl=https://redhat.cassandra.apache.org/50x/ 
gpgcheck=0 
repo_gpgcheck=0 
gpgkey=https://downloads.apache.org/cassandra/KEYS 
EOF

 dnf clean all && dnf makecache dnf install -y cassandra --skip-broken

Systemd Service Unit

Rather than auto-starting Cassandra, the script drops a custom cassandra-manager.service unit that explicitly sets JAVA_HOME to the Semeru installation. This prevents Cassandra from picking up an incorrect JVM:

cat <<'EOF' > /etc/systemd/system/cassandra-manager.service
[Unit]
Description=Apache Cassandra Lifecycle Manager
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
User=cassandra
Group=cassandra
Environment="JAVA_HOME=/opt/ibm/semeru"
Environment="PATH=/opt/ibm/semeru/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
ExecStart=/usr/sbin/cassandra -p /var/run/cassandra/cassandra.pid
PIDFile=/var/run/cassandra/cassandra.pid
RuntimeDirectory=cassandra
TimeoutStartSec=120
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload

The service is not enabled or auto-started — you decide when Cassandra runs.

---

Step 4 — Deploy

Initialize

terraform init

This downloads the IBM Cloud and null providers.

Plan

terraform plan -var-file="terraform.tfvars"

Review the output — you should see three resources being created: ibm_pi_network.public_net, ibm_pi_instance.cassandra_node, and null_resource.cassandra_init.

Apply

terraform apply -var-file="terraform.tfvars"

Total deployment time is approximately 10–15 minutes:

• Network creation: ~1 minute
• Instance boot: ~5 minutes
• init.sh execution (OS update + Java + Cassandra): ~5–10 minutes

---

Step 5 — Verify

Once terraform apply completes, grab the IP from the outputs:

terraform output instance_external_ip_address

SSH in as root:

ssh root@<external_ip>

Confirm IBM Semeru is the active JVM:

java -version
# Expected output:
# openjdk version "17.x.x" ...
# IBM Semeru Runtime Open Edition ...
# Eclipse OpenJ9 ...

Start Cassandra:

systemctl start cassandra-manager
systemctl status cassandra-manager

Once Cassandra is up (give it 30–60 seconds), verify the node is healthy:

nodetool status

You should see a single node listed as UN (Up, Normal):

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns   Host ID   Rack
UN  127.0.0.1  ...        16      100.0% <uuid>    rack1

To persist Cassandra across reboots:

systemctl enable cassandra-manager

---

Outputs Reference

Output	Description
instance_id	PowerVS LPAR instance ID
instance_ip_address	Private IP on the attached network
instance_external_ip_address	Public/external IP (used for SSH)

---

Security Considerations

• Credentials in terraform.tfvars — never commit this file. It is already listed in .gitignore.
• Sensitive variables — ibmcloud_api_key, rhsm_username, and rhsm_password are marked sensitive = true in variables.tf, so Terraform will not print them in plan or apply output.
• SSH agent forwarding — the provisioner connection uses agent = true, which relies on your local SSH agent. Make sure the private key matching ssh_key_name is loaded (ssh-add ~/.ssh/id_rsa).
• Firewall — the pub-vlan network exposes the instance publicly. Consider restricting inbound access to port 22 (SSH) and Cassandra’s native transport port 9042 to trusted CIDRs via IBM Cloud security groups.

---

Cleaning Up

To destroy all resources created by this deployment:

terraform destroy -var-file="terraform.tfvars"

This will remove the PowerVS instance and the public network. The RHEL 10 stock image is not deleted (it is a shared catalog image, not a user-imported resource).

---

Summary

This walkthrough showed how to:

1. Use Terraform with the IBM-Cloud/ibm provider to stand up a PowerVS LPAR on RHEL 10
2. Bootstrap the instance via SSH using a templated shell script
3. Install IBM Semeru Runtime (OpenJ9) for ppc64le — the right JVM for Power
4. Add the official Apache Cassandra 5.0 RPM repo and install Cassandra
5. Register a custom systemd service that pins JAVA_HOME to the Semeru installation

The full source is available in this repository — clone it, fill in your terraform.tfvars, and you’ll have Cassandra running on PowerVS in under 15 minutes.

Note the source is attached

By default, OpenShift is a greedy scheduler, when a Pod starts it grabs the first requested resource CPU/Memory. There is a reserved bit of memory and CPU for kubelet and System services saved. A Pod may end up spanning different NUMA domains

The CPU Manager packs each Pod onto NUMA domains:.

1. single-numa-node a pod might fail to schedule on a node that has enough total CPU capacity but lacks enough contiguous cores. If you go to schedule a 8 vCPU pod, and you have 4vcpu on one node, and 4 vcpu on another, it won’t be able to allocate ending with a TopologyAffinityError.
2. restricted packs it into as few NUMA nodes as possible. This one is probably preferred as it allows for rounding.

Keep in mind, this applies for the whole system, and you may want to isolate to only a few workers you can put them in a separate pool. You can see https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/scalability_and_performance/using-cpu-manager#setting_up_cpu_manager_using-cpu-manager-and-topology-manager

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: cpumanager-enabled
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    cpuManagerPolicy: static 
    cpuManagerReconcilePeriod: 5s
    topologyManagerPolicy: single-numa-node 
    reservedSystemCPUs: "0,1" 
    memoryManagerPolicy: Static

If a Pod is idle, those cores sit idle. They won’t share the spare cycles of another Pod.

If a Pod used 100m, it’ll need to specificy round numbers now cpu: "2" in some cases.

For large workloads , CPU pinning is most effective when the pod is sized to fit within a single NUMA node. If your pod is so large that it spans multiple sockets, the benefits of pinning diminish unless the application itself is NUMA-aware.

You can see more details at OpenShift 4.20: Using CPU Manager and Topology Manager [docs.redhat.com]

One of the more common challenges in disconnected or tightly controlled on-prem OpenShift environments is certificate validation when the cluster doesn’t have direct access to external DNS resolvers.

If cert-manager’s DNS-01 validation cannot reach the appropriate DNS infrastructure, certificate issuance may fail even when the DNS records themselves are correct. In these environments, you can configure the cert-manager operator to use specific internal recursive DNS servers instead of relying on default resolver behavior.

The following patch updates the cert-manager configuration to query only the designated recursive nameservers:

oc patch certmanager cluster --type=merge -p='{
  "spec": {
    "controllerConfig": {
      "overrideArgs": [
        "--dns01-recursive-nameservers=10.0.10.4:53,10.0.10.5:53",
        "--dns01-recursive-nameservers-only"
      ]
    }
  }
}'

This approach is particularly useful in air-gapped, restricted-network, or enterprise environments where outbound DNS traffic is blocked and all name resolution must flow through approved internal resolvers.

A small change, but often the difference between endless certificate troubleshooting and a clean, automated issuance process.

Co-authored and reposted here…

Kaushik Talathi, Engineer, OpenShift on IBM Power

Paul Bastide, Engineering Lead, OpenShift on IBM Power

When a worker node panics, the kernel’s last words are the most valuable debugging artifact you’ll ever get — if you managed to capture them. On your OpenShift Container Platform on IBM Power or on IBM Power Virtual Server, you can capture kernel crash dumps to available persistent memory (pmem) on the nodes. kdump is able to write vmcores to a pmem-backed filesystem: it’s fast, it’s dedicated, and it keeps multi-gigabyte crash dumps off the root disk or NFS servers.

Using this document, you learn how to: create a pmem namespace with ndctl, format and mount it via MachineConfig systemd units, configure kdump through butane, trigger a crash with echo c > /proc/sysrq-trigger — and the vmcore landed exactly where it should, on /var/mnt/pmem.

Red Hat CoreOS is an immutable OS which doesn’t shift with ndctl; you can’t just use it by default. This document shows how to use a privileged pod with access to /dev and /sys, running ndctl create-namespace --mode=fsdax against the host’s NVDIMM subsystem. This day-2 action is useful when troubleshooting and not intended to be used by default – in exceptional debugging of storage problems.

When we don’t have direct access to tools on the file system, we use oneshot systemd unit, delivered by MachineConfig, that runs the privileged container to execute:

[Unit]
Description=Create PMEM namespace (ndctl via privileged container)
ConditionPathExistsGlob=/sys/bus/nd/devices/region*
ConditionPathExists=!/dev/pmem0
Wants=network-online.target
After=network-online.target
Before=format-pmem0.service var-mnt-pmem.mount

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStartSec=600
ExecStart=/usr/local/bin/pmem-namespace.sh

[Install]
WantedBy=multi-user.target

The two Condition lines are doing the heavy lifting. If /dev/pmem0 already exists, there’s nothing to do. If the node has no NVDIMM regions at all, there’s nothing that can be done. In both cases systemd skips the unit — skipped, not failed — and boot proceeds normally. That’s the behavior we want across a fleet where not every worker has pmem.

The script it calls is small and idempotent:

#!/usr/bin/env bash
set -euo pipefail

NDCTL_IMAGE="${NDCTL_IMAGE:-quay.io/centos/centos:stream10}"
AUTHFILE="/var/lib/kubelet/config.json"

[ -b /dev/pmem0 ] && { echo "already present"; exit 0; }
ls /sys/bus/nd/devices/region* >/dev/null 2>&1 || { echo "no NVDIMM regions"; exit 0; }

AUTH_ARGS=()
[ -f "${AUTHFILE}" ] && AUTH_ARGS=(--authfile "${AUTHFILE}")

podman run --rm --privileged "${AUTH_ARGS[@]}" \
  -v /dev:/dev -v /sys:/sys \
  "${NDCTL_IMAGE}" \
  bash -c 'command -v ndctl >/dev/null 2>&1 || dnf -y -q install ndctl; ndctl create-namespace --mode=fsdax'

udevadm settle
[ -b /dev/pmem0 ] || { echo "expected /dev/pmem0" >&2; exit 1; }

By default it pulls CentOS Stream 10 and installs ndctl on the fly. For disconnected environments or faster boots, prebuild a two-line image (FROM quay.io/centos/centos:stream10 + dnf -y install ndctl daxctl), push it to your registry, and point NDCTL_IMAGE at it — the script already passes the node’s pull secret via --authfile.

The startup flows:

pmem-namespace.service  →  format-pmem0.service  →  var-mnt-pmem.mount  →  kdump.service
   (create if needed)        (mkfs once, ever)         (xfs, nofail)        (arm capture kernel)

All of it — the script, the units, the crashkernel= kernel arguments, /etc/kdump.conf, and /etc/sysconfig/kdump — lives in one butane file rendered to a single MachineConfig. The full listing is in the companion setup document; render and apply is the usual two-liner:

butane 99-worker-pmem-kdump.bu -o 99-worker-pmem-kdump.yaml
oc apply -f 99-worker-pmem-kdump.yaml

What to do with the cores

Dumps arrive under /var/mnt/pmem/<ip>-<date>-<time>/ as vmcore, vmcore-dmesg.txt, and kexec-dmesg.log. Three habits worth adopting:

Read vmcore-dmesg.txt first. It’s the crashed kernel’s ring buffer as plain text, panic backtrace at the bottom. A large fraction of crashes are diagnosed from this file alone, no tooling required.

Use toolbox for deep analysis. This is where toolbox shines. From oc debug node/<node> → chroot /host → toolbox, you get a privileged support-tools container where you can dnf install crash plus the matching kernel-debuginfo and open the vmcore in place. Debuginfo is huge and must match the crashed kernel exactly, so in practice we often stream the dump off the node instead —

oc debug node/worker-0 -- \
  tar czf - -C /host/var/mnt/pmem 127.0.0.1-2026-07-09-12:06:09 > vmcore-worker-0.tgz

— and run crash on a RHEL box where the right debuginfo installs cleanly.

Prune automatically. Persistent memory is not infinite, and a filtered vmcore is still roughly proportional to RAM. A tiny systemd timer in the same MachineConfig deletes dump directories older than 14 days, and its ConditionPathIsMountPoint=/var/mnt/pmem means it too is a no-op on nodes without pmem.

Verifying it

After the MachineConfigPool converges:

oc debug node/worker-0
chroot /host
systemctl status kdump
cat /sys/kernel/kexec_crash_loaded    # 1 means armed
findmnt /var/mnt/pmem                 # xfs on /dev/pmem0

And in a maintenance window, once you reboot, you can see the vmcore survived so you don’t have to wonder:

echo c > /proc/sysrq-trigger
# node dumps, reboots, and then:
find /var/mnt/pmem -type f
/var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore
/var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore-dmesg.txt
/var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/kexec-dmesg.log

One caveat for mixed fleets: on a node where the pmem mount never exists, kdump’s path /var/mnt/pmem falls back to the root filesystem. If only some of your workers carry pmem, put this MachineConfig on a dedicated pool (e.g., a worker-pmem role), or switch kdump.conf to a direct dump target (xfs /dev/pmem0 + path /) so kdump fails loudly rather than dumping to /.

Wrapping up

The pattern here generalizes beyond pmem: when RHCOS doesn’t have the tool, a privileged container run by podman from a oneshot systemd unit is how you run it at boot — with Condition* directives making the whole thing safe on nodes where it doesn’t apply. Toolbox is the interactive face of the same idea, and it earns its keep again when it’s time to open the vmcore with crash.

Credit where it’s due: the namespace-creation discovery, the kdump-to-pmem configuration, and the end-to-end crash validation are Kaushik Talathi’s work — this post just wires his findings into the boot sequence so nobody ever has to remember the privileged pod again.

Validated on OpenShift 4.18, RHCOS (RHEL 9.x, kernel 5.14.0-570.x).

References

• ndctl User Guide — Managing Namespaces
• ndctl-create-namespace(1)
• OpenShift Support Guide — gathering data and toolbox
• RHCOS debugging quick start with toolbox