Blog

  • Deploying Apache Cassandra on IBM PowerVS with Terraform and RHEL 10

    IBM Power Systems Virtual Server (PowerVS) is a great fit for data-intensive workloads. In this post, we’ll walk through deploying a single-node Apache Cassandra 5.0 instance on PowerVS using a RHEL 10 Bring-Your-Own-License (BYOL) image, fully automated with Terraform and a cloud-init shell script.

    By the end you’ll have:

    • A running PowerVS LPAR (s1022) on RHEL 10
    • IBM Semeru Runtime (OpenJ9) for Java on ppc64le
    • Apache Cassandra 5.0 installed and ready to start
    • A public network interface for SSH access

    Architecture Overview

    The deployment is split into two layers:

    Infrastructure (Terraform)

    ResourcePurpose
    ibm_pi_networkCreates a public pub-vlan network for external access
    ibm_pi_instanceProvisions the LPAR using the RHEL 10 BYOL stock image
    null_resourceSSHes into the instance after boot and runs the init script

    Configuration (init.sh)

    Once the LPAR is up, Terraform copies a templated init.sh script to the instance and executes it over SSH. The script:

    1. Registers the OS with Red Hat Subscription Manager (RHSM)
    2. Enables the RHEL 10 ppc64le base and appstream repos
    3. Updates the OS and installs tooling (wgetcurljq, etc.)
    4. Downloads and installs IBM Semeru Java 17 LTS for ppc64le from GitHub Releases
    5. Registers the cassandra YUM repo and installs Apache Cassandra 5.0
    6. Drops a cassandra-manager.service systemd unit (installed but not auto-started — you control when it runs)

    Prerequisites

    Before you begin, make sure you have:

    • An IBM Cloud account with PowerVS permissions
    • pre-provisioned PowerVS workspace (get the workspace GUID from the UI)
    • PowerVS SSH key registered in the workspace
    • Red Hat credentials (RHSM username and password) for a RHEL subscription
    • Terraform ≥ 1.3 installed locally
    • The IBM Cloud CLI with the PowerVS plugin (ibmcloud plugin install power-iaas)

    RHEL 10 BYOL image: This example uses the stock catalog image ID 585ca713-f303-45f0-a836-e9dd4f4c8f3b (RHEL10-BYOL) which is already available in PowerVS — no COS bucket or manual image upload required.


    Repository Layout

    .
    ├── main.tf                  # Provider, network, instance, and provisioner
    ├── variables.tf             # Input variable declarations
    ├── outputs.tf               # Instance ID and IP outputs
    ├── init.sh                  # cloud-init / SSH-executed bootstrap script
    └── terraform.tfvars.example # Template — copy to terraform.tfvars
    

    Step 1 — Configure Your Variables

    Copy the example variables file and fill in your values:

    cp terraform.tfvars.example terraform.tfvars
    

    Then edit terraform.tfvars:

    ibmcloud_api_key      = "YOUR_IBM_CLOUD_API_KEY"
    workspace_id          = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    region                = "us-south"
    zone                  = "dal12"
    ssh_key_name          = "my-powervs-ssh-key"
    ssh_private_key_path  = "~/.ssh/id_rsa"
    memory                = 16
    processors            = 2
    rhsm_username         = "your-redhat-username"
    rhsm_password         = "your-redhat-password"
    

    Never commit terraform.tfvars to source control — it contains your IBM Cloud API key and RHSM credentials. The .gitignore in this repo already excludes it, and all three sensitive variables are declared with sensitive = true in variables.tf so Terraform redacts them from plan/apply output.

    Variable Reference

    VariableDescriptionDefault
    ibmcloud_api_keyIBM Cloud API key
    workspace_idPowerVS workspace GUID
    regionIBM Cloud region (e.g. us-south)
    zoneIBM Cloud zone (e.g. dal12)
    ssh_key_nameSSH key registered in the PowerVS workspace
    ssh_private_key_pathLocal path to the matching private key~/.ssh/id_rsa
    memoryInstance memory in GiB16
    processorsNumber of virtual processors2
    rhsm_usernameRed Hat Subscription Manager username
    rhsm_passwordRed Hat Subscription Manager password

    Step 2 — Understand the Terraform Configuration

    Provider

    main.tf pins the IBM Cloud provider at ~> 2.4.0 and the HashiCorp null provider for the remote-exec provisioner:

    terraform {
      required_providers {
        ibm = {
          source  = "IBM-Cloud/ibm"
          version = "~> 2.4.0"
        }
        null = {
          source  = "hashicorp/null"
          version = "~> 3.0"
        }
      }
    }
    
    provider "ibm" {
      ibmcloud_api_key = var.ibmcloud_api_key
      region           = var.region
      zone             = var.zone
    }
    

    Public Network

    pub-vlan network is created to give the instance an external IP for SSH access:

    resource "ibm_pi_network" "public_net" {
      pi_cloud_instance_id = var.workspace_id
      pi_network_name      = "cassandra-public-net"
      pi_network_type      = "pub-vlan"
    }
    

    PowerVS Instance

    The LPAR is an s1022 system (Power10) using a shared processor, the stock RHEL 10 BYOL image, and attached to the public network:

    resource "ibm_pi_instance" "cassandra_node" {
      pi_cloud_instance_id = var.workspace_id
      pi_memory            = var.memory
      pi_processors        = var.processors
      pi_instance_name     = "cassandra-rhel10"
      pi_proc_type         = "shared"
      pi_sys_type          = "s1022"
      pi_image_id          = "585ca713-f303-45f0-a836-e9dd4f4c8f3b"
      pi_key_pair_name     = var.ssh_key_name
      pi_health_status     = "WARNING"
    
      pi_network {
        network_id = ibm_pi_network.public_net.network_id
      }
    
      depends_on = [ibm_pi_network.public_net]
    }
    

    pi_health_status = "WARNING" lets Terraform proceed even while the instance is still booting, which is normal — the null_resource provisioner handles the wait via SSH.

    Remote Provisioner

    After the instance is up, Terraform templates init.sh with your RHSM credentials and copies it over SSH, then executes it:

    resource "null_resource" "cassandra_init" {
      triggers = {
        instance_id = ibm_pi_instance.cassandra_node.instance_id
        script_hash = filemd5("${path.module}/init.sh")
      }
    
      connection {
        type    = "ssh"
        user    = "root"
        host    = ibm_pi_instance.cassandra_node.pi_network[0].external_ip
        agent   = true
        timeout = "10m"
      }
    
      provisioner "file" {
        content = templatefile("${path.module}/init.sh", {
          rhsm_username = var.rhsm_username
          rhsm_password = var.rhsm_password
        })
        destination = "/root/init.sh"
      }
    
      provisioner "remote-exec" {
        inline = [
          "chmod +x /root/init.sh",
          "bash /root/init.sh",
        ]
      }
    }
    

    The script_hash trigger means Terraform will re-run the provisioner if you change init.sh, which is handy during development.


    Step 3 — The Bootstrap Script (init.sh)

    init.sh is a Bash script executed on the instance as root. Here’s what it does, step by step.

    RHSM Registration

    subscription-manager register \
      --username="${rhsm_username}" \
      --password="${rhsm_password}" \
      --force
    
    subscription-manager repos \
      --enable=rhel-10-for-ppc64le-baseos-rpms \
      --enable=rhel-10-for-ppc64le-appstream-rpms
    

    The --force flag handles re-registration gracefully if the instance was previously registered. The correct repo slugs for RHEL 10 on Power LE (ppc64le) are rhel-10-for-ppc64le-*.

    OS Update and Tooling

    dnf update -y
    dnf install -y wget tar gzip jq curl
    

    IBM Semeru Java 17 (ppc64le)

    Cassandra requires Java, and IBM Semeru Runtime (OpenJ9) is the recommended JVM for Power. The script fetches the latest release URL dynamically from the GitHub API:

    SEMERU_LATEST_URL=$(curl -s https://api.github.com/repos/ibmruntimes/semeru17-binaries/releases/latest \
      | jq -r '.assets[] | select(.name | contains("jdk_ppc64le_linux")) | select(.name | endswith(".tar.gz")) | .browser_download_url')
    
    wget -qO /tmp/semeru.tar.gz "$SEMERU_LATEST_URL"
    mkdir -p /opt/ibm/semeru
    tar -xzf /tmp/semeru.tar.gz -C /opt/ibm/semeru --strip-components=1
    rm -f /tmp/semeru.tar.gz
    
    alternatives --install /usr/bin/java java /opt/ibm/semeru/bin/java 1
    echo "export JAVA_HOME=/opt/ibm/semeru" > /etc/profile.d/semeru.sh
    export JAVA_HOME=/opt/ibm/semeru
    export PATH="$JAVA_HOME/bin:$PATH"
    

    Installing under /opt/ibm/semeru keeps it clean and separate from the system Java. The alternatives registration makes java available system-wide.

    Apache Cassandra 5.0

    The official Apache Cassandra RPM repo is added and the package installed. --skip-broken is used because Cassandra’s bundled JVM dependency is skipped in favour of Semeru:

    rpm --import https://downloads.apache.org/cassandra/KEYS
    
    cat <<'EOF' > /etc/yum.repos.d/cassandra.repo
    [cassandra]
    name=Apache Cassandra 
    baseurl=https://redhat.cassandra.apache.org/50x/ 
    gpgcheck=0 
    repo_gpgcheck=0 
    gpgkey=https://downloads.apache.org/cassandra/KEYS 
    EOF
    
     dnf clean all && dnf makecache dnf install -y cassandra --skip-broken

    Systemd Service Unit

    Rather than auto-starting Cassandra, the script drops a custom cassandra-manager.service unit that explicitly sets JAVA_HOME to the Semeru installation. This prevents Cassandra from picking up an incorrect JVM:

    cat <<'EOF' > /etc/systemd/system/cassandra-manager.service
    [Unit]
    Description=Apache Cassandra Lifecycle Manager
    After=network-online.target
    Wants=network-online.target
    
    [Service]
    Type=forking
    User=cassandra
    Group=cassandra
    Environment="JAVA_HOME=/opt/ibm/semeru"
    Environment="PATH=/opt/ibm/semeru/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
    ExecStart=/usr/sbin/cassandra -p /var/run/cassandra/cassandra.pid
    PIDFile=/var/run/cassandra/cassandra.pid
    RuntimeDirectory=cassandra
    TimeoutStartSec=120
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target
    EOF
    
    systemctl daemon-reload
    

    The service is not enabled or auto-started — you decide when Cassandra runs.


    Step 4 — Deploy

    Initialize

    terraform init
    

    This downloads the IBM Cloud and null providers.

    Plan

    terraform plan -var-file="terraform.tfvars"
    

    Review the output — you should see three resources being created: ibm_pi_network.public_netibm_pi_instance.cassandra_node, and null_resource.cassandra_init.

    Apply

    terraform apply -var-file="terraform.tfvars"
    

    Total deployment time is approximately 10–15 minutes:

    • Network creation: ~1 minute
    • Instance boot: ~5 minutes
    • init.sh execution (OS update + Java + Cassandra): ~5–10 minutes

    Step 5 — Verify

    Once terraform apply completes, grab the IP from the outputs:

    terraform output instance_external_ip_address
    

    SSH in as root:

    ssh root@<external_ip>
    

    Confirm IBM Semeru is the active JVM:

    java -version
    # Expected output:
    # openjdk version "17.x.x" ...
    # IBM Semeru Runtime Open Edition ...
    # Eclipse OpenJ9 ...
    

    Start Cassandra:

    systemctl start cassandra-manager
    systemctl status cassandra-manager
    

    Once Cassandra is up (give it 30–60 seconds), verify the node is healthy:

    nodetool status
    

    You should see a single node listed as UN (Up, Normal):

    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Tokens  Owns   Host ID   Rack
    UN  127.0.0.1  ...        16      100.0% <uuid>    rack1
    

    To persist Cassandra across reboots:

    systemctl enable cassandra-manager
    

    Outputs Reference

    OutputDescription
    instance_idPowerVS LPAR instance ID
    instance_ip_addressPrivate IP on the attached network
    instance_external_ip_addressPublic/external IP (used for SSH)

    Security Considerations

    • Credentials in terraform.tfvars — never commit this file. It is already listed in .gitignore.
    • Sensitive variables — ibmcloud_api_keyrhsm_username, and rhsm_password are marked sensitive = true in variables.tf, so Terraform will not print them in plan or apply output.
    • SSH agent forwarding — the provisioner connection uses agent = true, which relies on your local SSH agent. Make sure the private key matching ssh_key_name is loaded (ssh-add ~/.ssh/id_rsa).
    • Firewall — the pub-vlan network exposes the instance publicly. Consider restricting inbound access to port 22 (SSH) and Cassandra’s native transport port 9042 to trusted CIDRs via IBM Cloud security groups.

    Cleaning Up

    To destroy all resources created by this deployment:

    terraform destroy -var-file="terraform.tfvars"
    

    This will remove the PowerVS instance and the public network. The RHEL 10 stock image is not deleted (it is a shared catalog image, not a user-imported resource).


    Summary

    This walkthrough showed how to:

    1. Use Terraform with the IBM-Cloud/ibm provider to stand up a PowerVS LPAR on RHEL 10
    2. Bootstrap the instance via SSH using a templated shell script
    3. Install IBM Semeru Runtime (OpenJ9) for ppc64le — the right JVM for Power
    4. Add the official Apache Cassandra 5.0 RPM repo and install Cassandra
    5. Register a custom systemd service that pins JAVA_HOME to the Semeru installation

    The full source is available in this repository — clone it, fill in your terraform.tfvars, and you’ll have Cassandra running on PowerVS in under 15 minutes.

    Note the source is attached

  • Enabling CPU Manager and CPU Pinning in OpenShift

    By default, OpenShift is a greedy scheduler, when a Pod starts it grabs the first requested resource CPU/Memory. There is a reserved bit of memory and CPU for kubelet and System services saved. A Pod may end up spanning different NUMA domains

    The CPU Manager packs each Pod onto NUMA domains:.

    1. single-numa-node a pod might fail to schedule on a node that has enough total CPU capacity but lacks enough contiguous cores. If you go to schedule a 8 vCPU pod, and you have 4vcpu on one node, and 4 vcpu on another, it won’t be able to allocate ending with a TopologyAffinityError.
    2. restricted packs it into as few NUMA nodes as possible. This one is probably preferred as it allows for rounding.

    Keep in mind, this applies for the whole system, and you may want to isolate to only a few workers you can put them in a separate pool. You can see https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/scalability_and_performance/using-cpu-manager#setting_up_cpu_manager_using-cpu-manager-and-topology-manager

    apiVersion: machineconfiguration.openshift.io/v1
    kind: KubeletConfig
    metadata:
      name: cpumanager-enabled
    spec:
      machineConfigPoolSelector:
        matchLabels:
          pools.operator.machineconfiguration.openshift.io/worker: ""
      kubeletConfig:
        cpuManagerPolicy: static 
        cpuManagerReconcilePeriod: 5s
        topologyManagerPolicy: single-numa-node 
        reservedSystemCPUs: "0,1" 
        memoryManagerPolicy: Static
    

    If a Pod is idle, those cores sit idle. They won’t share the spare cycles of another Pod.

    If a Pod used 100m, it’ll need to specificy round numbers now cpu: "2" in some cases.

    For large workloads , CPU pinning is most effective when the pod is sized to fit within a single NUMA node. If your pod is so large that it spans multiple sockets, the benefits of pinning diminish unless the application itself is NUMA-aware.

    You can see more details at OpenShift 4.20: Using CPU Manager and Topology Manager [docs.redhat.com]

  • Workloads: From Bare Metal to KubeVirt

    Workloads run on full physical servers or are managed as virtual machines. KubeVirt brings cloud-native approach to fleet management and makes VMs first-class Kubernetes resources.

    1. Physical Infrastructure

    Typically, things begin with a physical server. A Baseboard Management Controller (BMC) provides hardware management, while firmware initializes CPU and memory resources. At this stage, all resources belong to a single system with no virtualization.

    Key point: Physical hardware only, with no abstraction layer.


    2. Traditional Virtualization

    A host operating system such as RHEL, SUSE, or Ubuntu runs on the server. Libvirt and QEMU provide virtualization services, allowing multiple VMs to share the same hardware.

    Key point: Virtual machines improve hardware utilization while remaining tied to individual hosts.


    3. Kubernetes + KubeVirt

    KubeVirt integrates virtualization into Kubernetes. Instead of managing VMs directly on each host, Kubernetes manages VM lifecycle through KubeVirt components such as Virt Operator, Virt Controller, and Virt Launcher.

    Virtual machines become Kubernetes-managed workloads that can be scheduled, automated, and operated alongside containers.


    Conclusion

    The progression is straightforward Physical Server run Libvirt/QEMU Virtualization and are managed as a fleet with Kubernetes-Orchestrated Virtualization with KubeVirt.

    Note: based on an ad-hoc presentation….

  • Aside: cert-manager: If You’re Running On-Prem Without Direct DNS Access

    One of the more common challenges in disconnected or tightly controlled on-prem OpenShift environments is certificate validation when the cluster doesn’t have direct access to external DNS resolvers.

    If cert-manager’s DNS-01 validation cannot reach the appropriate DNS infrastructure, certificate issuance may fail even when the DNS records themselves are correct. In these environments, you can configure the cert-manager operator to use specific internal recursive DNS servers instead of relying on default resolver behavior.

    The following patch updates the cert-manager configuration to query only the designated recursive nameservers:

    oc patch certmanager cluster --type=merge -p='{
      "spec": {
        "controllerConfig": {
          "overrideArgs": [
            "--dns01-recursive-nameservers=10.0.10.4:53,10.0.10.5:53",
            "--dns01-recursive-nameservers-only"
          ]
        }
      }
    }'
    

    This approach is particularly useful in air-gapped, restricted-network, or enterprise environments where outbound DNS traffic is blocked and all name resolution must flow through approved internal resolvers.

    A small change, but often the difference between endless certificate troubleshooting and a clean, automated issuance process.

  • Capturing Kernel Crash Dumps on Persistent Memory on OpenShift Container Platform on IBM Power

    Co-authored and reposted here…

    Kaushik Talathi, Engineer, OpenShift on IBM Power

    Paul Bastide, Engineering Lead, OpenShift on IBM Power

    When a worker node panics, the kernel’s last words are the most valuable debugging artifact you’ll ever get — if you managed to capture them. On your OpenShift Container Platform on IBM Power or on IBM Power Virtual Server, you can capture kernel crash dumps to available persistent memory (pmem) on the nodes. kdump is able to write vmcores to a pmem-backed filesystem: it’s fast, it’s dedicated, and it keeps multi-gigabyte crash dumps off the root disk or NFS servers.

    Using this document, you learn how to: create a pmem namespace with ndctl, format and mount it via MachineConfig systemd units, configure kdump through butane, trigger a crash with echo c > /proc/sysrq-trigger — and the vmcore landed exactly where it should, on /var/mnt/pmem.

    Red Hat CoreOS is an immutable OS which doesn’t shift with ndctl; you can’t just use it by default. This document shows how to use a privileged pod with access to /dev and /sys, running ndctl create-namespace --mode=fsdax against the host’s NVDIMM subsystem. This day-2 action is useful when troubleshooting and not intended to be used by default – in exceptional debugging of storage problems.

    When we don’t have direct access to tools on the file system, we use oneshot systemd unit, delivered by MachineConfig, that runs the privileged container to execute:

    [Unit]
    Description=Create PMEM namespace (ndctl via privileged container)
    ConditionPathExistsGlob=/sys/bus/nd/devices/region*
    ConditionPathExists=!/dev/pmem0
    Wants=network-online.target
    After=network-online.target
    Before=format-pmem0.service var-mnt-pmem.mount
    
    [Service]
    Type=oneshot
    RemainAfterExit=yes
    TimeoutStartSec=600
    ExecStart=/usr/local/bin/pmem-namespace.sh
    
    [Install]
    WantedBy=multi-user.target
    

    The two Condition lines are doing the heavy lifting. If /dev/pmem0 already exists, there’s nothing to do. If the node has no NVDIMM regions at all, there’s nothing that can be done. In both cases systemd skips the unit — skipped, not failed — and boot proceeds normally. That’s the behavior we want across a fleet where not every worker has pmem.

    The script it calls is small and idempotent:

    #!/usr/bin/env bash
    set -euo pipefail
    
    NDCTL_IMAGE="${NDCTL_IMAGE:-quay.io/centos/centos:stream10}"
    AUTHFILE="/var/lib/kubelet/config.json"
    
    [ -b /dev/pmem0 ] && { echo "already present"; exit 0; }
    ls /sys/bus/nd/devices/region* >/dev/null 2>&1 || { echo "no NVDIMM regions"; exit 0; }
    
    AUTH_ARGS=()
    [ -f "${AUTHFILE}" ] && AUTH_ARGS=(--authfile "${AUTHFILE}")
    
    podman run --rm --privileged "${AUTH_ARGS[@]}" \
      -v /dev:/dev -v /sys:/sys \
      "${NDCTL_IMAGE}" \
      bash -c 'command -v ndctl >/dev/null 2>&1 || dnf -y -q install ndctl; ndctl create-namespace --mode=fsdax'
    
    udevadm settle
    [ -b /dev/pmem0 ] || { echo "expected /dev/pmem0" >&2; exit 1; }
    

    By default it pulls CentOS Stream 10 and installs ndctl on the fly. For disconnected environments or faster boots, prebuild a two-line image (FROM quay.io/centos/centos:stream10 + dnf -y install ndctl daxctl), push it to your registry, and point NDCTL_IMAGE at it — the script already passes the node’s pull secret via --authfile.

    The startup flows:

    pmem-namespace.service  →  format-pmem0.service  →  var-mnt-pmem.mount  →  kdump.service
       (create if needed)        (mkfs once, ever)         (xfs, nofail)        (arm capture kernel)
    

    All of it — the script, the units, the crashkernel= kernel arguments, /etc/kdump.conf, and /etc/sysconfig/kdump — lives in one butane file rendered to a single MachineConfig. The full listing is in the companion setup document; render and apply is the usual two-liner:

    butane 99-worker-pmem-kdump.bu -o 99-worker-pmem-kdump.yaml
    oc apply -f 99-worker-pmem-kdump.yaml
    

    What to do with the cores

    Dumps arrive under /var/mnt/pmem/<ip>-<date>-<time>/ as vmcorevmcore-dmesg.txt, and kexec-dmesg.log. Three habits worth adopting:

    Read vmcore-dmesg.txt first. It’s the crashed kernel’s ring buffer as plain text, panic backtrace at the bottom. A large fraction of crashes are diagnosed from this file alone, no tooling required.

    Use toolbox for deep analysis. This is where toolbox shines. From oc debug node/<node> → chroot /host → toolbox, you get a privileged support-tools container where you can dnf install crash plus the matching kernel-debuginfo and open the vmcore in place. Debuginfo is huge and must match the crashed kernel exactly, so in practice we often stream the dump off the node instead —

    oc debug node/worker-0 -- \
      tar czf - -C /host/var/mnt/pmem 127.0.0.1-2026-07-09-12:06:09 > vmcore-worker-0.tgz
    

    — and run crash on a RHEL box where the right debuginfo installs cleanly.

    Prune automatically. Persistent memory is not infinite, and a filtered vmcore is still roughly proportional to RAM. A tiny systemd timer in the same MachineConfig deletes dump directories older than 14 days, and its ConditionPathIsMountPoint=/var/mnt/pmem means it too is a no-op on nodes without pmem.

    Verifying it

    After the MachineConfigPool converges:

    oc debug node/worker-0
    chroot /host
    systemctl status kdump
    cat /sys/kernel/kexec_crash_loaded    # 1 means armed
    findmnt /var/mnt/pmem                 # xfs on /dev/pmem0
    

    And in a maintenance window, once you reboot, you can see the vmcore survived so you don’t have to wonder:

    echo c > /proc/sysrq-trigger
    # node dumps, reboots, and then:
    find /var/mnt/pmem -type f
    /var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore
    /var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore-dmesg.txt
    /var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/kexec-dmesg.log
    

    One caveat for mixed fleets: on a node where the pmem mount never exists, kdump’s path /var/mnt/pmem falls back to the root filesystem. If only some of your workers carry pmem, put this MachineConfig on a dedicated pool (e.g., a worker-pmem role), or switch kdump.conf to a direct dump target (xfs /dev/pmem0 + path /) so kdump fails loudly rather than dumping to /.

    Wrapping up

    The pattern here generalizes beyond pmem: when RHCOS doesn’t have the tool, a privileged container run by podman from a oneshot systemd unit is how you run it at boot — with Condition* directives making the whole thing safe on nodes where it doesn’t apply. Toolbox is the interactive face of the same idea, and it earns its keep again when it’s time to open the vmcore with crash.

    Credit where it’s due: the namespace-creation discovery, the kdump-to-pmem configuration, and the end-to-end crash validation are Kaushik Talathi’s work — this post just wires his findings into the boot sequence so nobody ever has to remember the privileged pod again.

    Validated on OpenShift 4.18, RHCOS (RHEL 9.x, kernel 5.14.0-570.x).

    References

  • Aside: Applying Kernel Tunnables quickly (randomize_va_space)

    When a Tuned profile changes a sysctl such as:

    [sysctl]
    kernel.randomize_va_space=0

    You can use TuneD Custom Resource:

    apiVersion: tuned.openshift.io/v1
    kind: Tuned
    metadata:
      name: disable-aslr
      namespace: openshift-cluster-node-tuning-operator
    spec:
      profile:
      - name: disable-aslr-profile
        data: |
          [main]
          summary=Disable ASLR
          include=openshift-node
    
          [sysctl]
          kernel.randomize_va_space=0
    
      recommend:
      - priority: 20
        profile: disable-aslr-profile
        match:
        - label: node-role.kubernetes.io/worker

    The Node Tuning Operator/tuned applies it dynamically via the sysctl interface, so a reboot is typically not required. The new value takes effect immediately on the node after the profile is applied.

    You can verify:

    oc debug node/<node>
    chroot /host
    
    sysctl kernel.randomize_va_space

    or

    cat /proc/sys/kernel/randomize_va_space

    Disabling ASLR through a Tuned object, the setting is applied without a node reboot. You can usually observe the change within seconds after the profile becomes active.

  • Using zstd for Fast and Efficient Compression on OpenShift

    When you need to archive files directly on an OpenShift node, zstd is a great choice because it is already available on the system and provides an excellent balance of compression ratio and performance.

    Access the Target Node

    Start by opening a remote shell to the node:

    oc rsh node/<node-target>

    Maximum Compression

    To prioritize the smallest archive size, use the ultra compression level (-22) with all available CPU threads:

    time tar -I 'zstd -T0 --ultra -22' -cf /tmp/archive.tar.zst /tmp/random_100m.dat

    Example result:

    real    0m30.007s
    user    0m29.346s
    sys     0m0.384s

    Maximum Compression with a Single Thread

    If you want to limit CPU usage, run the same compression level with a single thread:

    time tar -I 'zstd -T1 --ultra -22' -cf /tmp/archive.tar.zst /tmp/random_100m.dat

    Example result:

    real    0m30.677s
    user    0m30.322s
    sys     0m0.275s

    The runtime is similar, making this a viable option when CPU resources are constrained.

    Fast Throughput Compression

    For large files where speed is more important than archive size, use a lower compression level:

    time tar -I 'zstd -T0 -1' -cf /tmp/archive.tar.zst /var/log/random_1000m.dat

    Example result:

    real    0m18.716s
    user    0m14.327s
    sys     0m16.420s

    Fast Compression with One Thread

    You can further reduce CPU consumption by limiting compression to a single thread:

    time tar -I 'zstd -T1 -1' -cf /tmp/archive.tar.zst /var/log/random_1000m.dat

    Example result:

    real    0m19.121s
    user    0m13.434s
    sys     0m14.402s

    Key Takeaways

    • Use --ultra -22 when achieving the highest compression ratio is the primary goal.
    • Use -1 for the fastest archive creation and best throughput.
    • -T0 allows zstd to use all available CPU threads.
    • -T1 limits compression to a single thread and can be useful on busy systems.
    • In these examples, single-threaded and multi-threaded runs showed only minor differences in elapsed time, making thread selection largely a resource-management decision.
  • OpenShift: Adding Public Key to core user

    Post-installation, the supported way to add another SSH public key for the core user on OpenShift/RHCOS nodes is by updating or creating a MachineConfig that contains all desired sshAuthorizedKeys. The Machine Config Operator (MCO) then rolls the change out to the targeted MachineConfigPool.

    Note: OpenShift commonly uses 99-worker-ssh and 99-master-ssh to manage SSH keys for the core user.

    1. Export the existing MachineConfig for worker
    oc get machineconfig 99-worker-ssh -o yaml > 99-worker-ssh.yaml
    1. Add the second public key

    Edit the file and ensure both keys are present under sshAuthorizedKeys.

    Example:

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
      name: 99-worker-ssh
    spec:
      config:
        ignition:
          version: 3.5.0
        passwd:
          users:
          - name: core
            sshAuthorizedKeys:
            - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAEXISTINGKEY user1@example
            - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAANEWKEY user2@example

    Important: The sshAuthorizedKeys list is authoritative. Include the existing key(s) plus the new key. Do not provide only the new key unless you want to remove the old ones. The MCO only supports modifying sshAuthorizedKeys for the core user.

    1. Apply the MachineConfig
    oc apply -f 99-worker-ssh.yaml

    The Machine Config Operator will render a new configuration and update the nodes in the associated MachineConfigPool.

    1. Watch the rollout

    Check the MachineConfigPool:

    oc get mcp

    Watch until the pool reports:

    • UPDATED=True
    • UPDATING=False
    • DEGRADED=False
    1. Verify one of the nodes
    ssh core@<node-ip>

    Using either the original key or the newly added key.

    You can also verify on the node:

    oc debug node/<node-name>
    chroot /host
    cat /home/core/.ssh/authorized_keys

    The file should contain both public keys

    This will add both public keys to the core user on all worker nodes.

    You can repeat for the control plane.

    References: Update SSH Keys

  • IPI PowerVS: Power10 Migration in One Change

    Running an OpenShift IPI cluster on PowerVS? Moving to Power10 can be as simple as updating the machine systemType and letting OpenShift perform a rolling node replacement. The cluster provisions new nodes, migrates workloads, and removes the old VMs automatically. Full details at Christy Norman’s blog. [community.ibm.com]

    Control plane

    oc patch controlplanemachineset cluster \
      -n openshift-machine-api \
      --type=merge \
      -p '{"spec":{"template":{"machines_v1beta1_machine_openshift_io":{"spec":{"providerSpec":{"value":{"systemType":"s1022"}}}}}}}'
    

    Workers

    oc patch machineset <worker-machineset> \
      -n openshift-machine-api \
      --type=merge \
      -p '{"spec":{"template":{"spec":{"providerSpec":{"value":{"systemType":"s1022"}}}}}}'
    

    Credit: Christy Norman, “Migrating an OpenShift IPI Cluster to Power10 on Power Virtual Server.” [community.ibm.com]

  • OpenShift 4.22.0 ClusterImagePolicy Feature enforces signature verification on Release and Payload

    With OpenShift Container Platform 4.22, Red Hat enforces signature verification for the release image and payload (cluster operators and oeprands). The verification is controlled by the ClusterImagePolicy which specifies the Root CA and a scope of quay.io/openshift-release-dev/ocp-release and quay.io/openshift-release-dev/ocp-v4.0-art-dev. When an image is used on a node, the signatures are pulled from the mirror along with the image, and verified before starting a container. This verification expands on the ocp-release scope.

    apiVersion: config.openshift.io/v1
    kind: ClusterImagePolicy
    metadata:
      name: openshift
    spec:
      policy:
        rootOfTrust:
          policyType: PublicKey
          publicKey:
            keyData: ...
      scopes:
      - quay.io/openshift-release-dev/ocp-release
      - quay.io/openshift-release-dev/ocp-v4.0-art-dev
      ...
    

    If you are running a disconnected cluster and the signature is missing from your mirror, you’ll see the following:

    • Install: The bootstrap node hangs and does not install as the signature is not verified.
    • Upgrade: The ClusterImagePolicy enforces signature verification. If the signatures are missing from your mirror, the Cluster Version Operator (CVO) will be blocked, preventing node updates.

    In order to continue, you may do one of the following:

    1. Use oc mirror --v2 to mirror your content. This feature automatically honors signatures … see Mirroring images for a disconnected installation using the oc-mirror plugin
    2. If you are currently using oc adm release mirror, you can copy the sig file for the release payload:
    $ oc image mirror quay.io/openshift-release-dev/ocp-release:${RELEASE_DIGEST}.sig registry.example.com/openshift/whatever:${RELEASE_DIGEST}.sig
    

    You repeat for each image listed in the release.txt

    RELEASE_DIGEST:: Specifies your digest image with the : character replaced by a - character. For example: sha256:884e1ff5effeaa04467fab9725900e7f0ed1daa89a7734644f14783014cebdee becomes sha256-884e1ff5effeaa04467fab9725900e7f0ed1daa89a7734644f14783014cebdee.sig.

    You must switch to using oc mirror --v2.

    Good luck with your disconnected clusters, and ensure image signatures are present in your local mirror using one of the mirroring methods.

    Note: you can use this with your application / deliverable for enhanced security.

    References

    1. Red Hat OpenShift Docs: Chapter 12. Manage secure signatures with sigstore
    2. Red Hat Developer: How to verify container signatures in disconnected OpenShift
    3. Red Hat Developer: Verify Cosign bring-your-own PKI signature on OpenShift
    4. IBM Power Blog Mirroring for OpenShift on IBM Power in Disconnected Environments
    5. IBM Power Blog OpenShift 4.21.0 ClusterImagePolicy Feature enforces signature verification