Co-authored and reposted here…
Kaushik Talathi, Engineer, OpenShift on IBM Power
Paul Bastide, Engineering Lead, OpenShift on IBM Power
When a worker node panics, the kernel’s last words are the most valuable debugging artifact you’ll ever get — if you managed to capture them. On your OpenShift Container Platform on IBM Power or on IBM Power Virtual Server, you can capture kernel crash dumps to available persistent memory (pmem) on the nodes. kdump is able to write vmcores to a pmem-backed filesystem: it’s fast, it’s dedicated, and it keeps multi-gigabyte crash dumps off the root disk or NFS servers.
Using this document, you learn how to: create a pmem namespace with ndctl, format and mount it via MachineConfig systemd units, configure kdump through butane, trigger a crash with echo c > /proc/sysrq-trigger — and the vmcore landed exactly where it should, on /var/mnt/pmem.
Red Hat CoreOS is an immutable OS which doesn’t shift with ndctl; you can’t just use it by default. This document shows how to use a privileged pod with access to /dev and /sys, running ndctl create-namespace --mode=fsdax against the host’s NVDIMM subsystem. This day-2 action is useful when troubleshooting and not intended to be used by default – in exceptional debugging of storage problems.
When we don’t have direct access to tools on the file system, we use oneshot systemd unit, delivered by MachineConfig, that runs the privileged container to execute:
[Unit]
Description=Create PMEM namespace (ndctl via privileged container)
ConditionPathExistsGlob=/sys/bus/nd/devices/region*
ConditionPathExists=!/dev/pmem0
Wants=network-online.target
After=network-online.target
Before=format-pmem0.service var-mnt-pmem.mount
[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStartSec=600
ExecStart=/usr/local/bin/pmem-namespace.sh
[Install]
WantedBy=multi-user.target
The two Condition lines are doing the heavy lifting. If /dev/pmem0 already exists, there’s nothing to do. If the node has no NVDIMM regions at all, there’s nothing that can be done. In both cases systemd skips the unit — skipped, not failed — and boot proceeds normally. That’s the behavior we want across a fleet where not every worker has pmem.
The script it calls is small and idempotent:
#!/usr/bin/env bash
set -euo pipefail
NDCTL_IMAGE="${NDCTL_IMAGE:-quay.io/centos/centos:stream10}"
AUTHFILE="/var/lib/kubelet/config.json"
[ -b /dev/pmem0 ] && { echo "already present"; exit 0; }
ls /sys/bus/nd/devices/region* >/dev/null 2>&1 || { echo "no NVDIMM regions"; exit 0; }
AUTH_ARGS=()
[ -f "${AUTHFILE}" ] && AUTH_ARGS=(--authfile "${AUTHFILE}")
podman run --rm --privileged "${AUTH_ARGS[@]}" \
-v /dev:/dev -v /sys:/sys \
"${NDCTL_IMAGE}" \
bash -c 'command -v ndctl >/dev/null 2>&1 || dnf -y -q install ndctl; ndctl create-namespace --mode=fsdax'
udevadm settle
[ -b /dev/pmem0 ] || { echo "expected /dev/pmem0" >&2; exit 1; }
By default it pulls CentOS Stream 10 and installs ndctl on the fly. For disconnected environments or faster boots, prebuild a two-line image (FROM quay.io/centos/centos:stream10 + dnf -y install ndctl daxctl), push it to your registry, and point NDCTL_IMAGE at it — the script already passes the node’s pull secret via --authfile.
The startup flows:
pmem-namespace.service → format-pmem0.service → var-mnt-pmem.mount → kdump.service
(create if needed) (mkfs once, ever) (xfs, nofail) (arm capture kernel)
All of it — the script, the units, the crashkernel= kernel arguments, /etc/kdump.conf, and /etc/sysconfig/kdump — lives in one butane file rendered to a single MachineConfig. The full listing is in the companion setup document; render and apply is the usual two-liner:
butane 99-worker-pmem-kdump.bu -o 99-worker-pmem-kdump.yaml
oc apply -f 99-worker-pmem-kdump.yaml
What to do with the cores
Dumps arrive under /var/mnt/pmem/<ip>-<date>-<time>/ as vmcore, vmcore-dmesg.txt, and kexec-dmesg.log. Three habits worth adopting:
Read vmcore-dmesg.txt first. It’s the crashed kernel’s ring buffer as plain text, panic backtrace at the bottom. A large fraction of crashes are diagnosed from this file alone, no tooling required.
Use toolbox for deep analysis. This is where toolbox shines. From oc debug node/<node> → chroot /host → toolbox, you get a privileged support-tools container where you can dnf install crash plus the matching kernel-debuginfo and open the vmcore in place. Debuginfo is huge and must match the crashed kernel exactly, so in practice we often stream the dump off the node instead —
oc debug node/worker-0 -- \
tar czf - -C /host/var/mnt/pmem 127.0.0.1-2026-07-09-12:06:09 > vmcore-worker-0.tgz
— and run crash on a RHEL box where the right debuginfo installs cleanly.
Prune automatically. Persistent memory is not infinite, and a filtered vmcore is still roughly proportional to RAM. A tiny systemd timer in the same MachineConfig deletes dump directories older than 14 days, and its ConditionPathIsMountPoint=/var/mnt/pmem means it too is a no-op on nodes without pmem.
Verifying it
After the MachineConfigPool converges:
oc debug node/worker-0
chroot /host
systemctl status kdump
cat /sys/kernel/kexec_crash_loaded # 1 means armed
findmnt /var/mnt/pmem # xfs on /dev/pmem0
And in a maintenance window, once you reboot, you can see the vmcore survived so you don’t have to wonder:
echo c > /proc/sysrq-trigger
# node dumps, reboots, and then:
find /var/mnt/pmem -type f
/var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore
/var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore-dmesg.txt
/var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/kexec-dmesg.log
One caveat for mixed fleets: on a node where the pmem mount never exists, kdump’s path /var/mnt/pmem falls back to the root filesystem. If only some of your workers carry pmem, put this MachineConfig on a dedicated pool (e.g., a worker-pmem role), or switch kdump.conf to a direct dump target (xfs /dev/pmem0 + path /) so kdump fails loudly rather than dumping to /.
Wrapping up
The pattern here generalizes beyond pmem: when RHCOS doesn’t have the tool, a privileged container run by podman from a oneshot systemd unit is how you run it at boot — with Condition* directives making the whole thing safe on nodes where it doesn’t apply. Toolbox is the interactive face of the same idea, and it earns its keep again when it’s time to open the vmcore with crash.
Credit where it’s due: the namespace-creation discovery, the kdump-to-pmem configuration, and the end-to-end crash validation are Kaushik Talathi’s work — this post just wires his findings into the boot sequence so nobody ever has to remember the privileged pod again.
Validated on OpenShift 4.18, RHCOS (RHEL 9.x, kernel 5.14.0-570.x).
References
