Blog

  • Using an EgressIP with UserDefinedNetwork on OpenShift Container Platform for IBM Power

    As OpenShift Container Platform (OCP) adoption continues to grow on IBM Power systems, customers are increasingly looking to combine advanced networking capabilities with on-premises PowerVM deployments. One area that often generates questions is how EgressIP behaves with a primary UserDefinedNetwork (UDN) and how traffic is routed from workloads attached to that network.

    This walkthrough demonstrates how to deploy a Layer 2 primary UserDefinedNetwork, assign an EgressIP, and validate that outbound traffic from a pod traverses the UDN interface rather than the cluster default network. The example is particularly relevant for OpenShift on IBM Power, including PowerVM environments managed through PowerVC.

    Why UserDefinedNetworks Matter

    UserDefinedNetworks (UDNs) allow OpenShift administrators to create custom pod networking domains separate from the cluster’s default network. When deployed as a Primary network, the UDN becomes the pod’s default network interface and routing domain.

    This can be useful for:

    • Application isolation
    • Dedicated routing paths
    • Network segmentation
    • Multi-tenant environments
    • Advanced EgressIP scenarios

    For organizations running OCP on IBM Power, UDNs provide the same flexibility available on other supported OpenShift platforms while leveraging the scale and resiliency of Power infrastructure.

    Prerequisites

    Before creating the EgressIP resource, choose a valid IP address that:

    • Is currently unused
    • Is routable from your worker nodes
    • Exists on the same Layer 2 segment as the node interface hosting the EgressIP

    ⚠️ Replace EGRESS_IP_ADDRESS with a valid address from your environment.

    For on-premises PowerVM deployments, this IP should belong to the same network segment as the worker nodes that will advertise the EgressIP.

    Create the UDN-Enabled Namespace

    Namespaces that use a primary UDN must be labeled appropriately.

    cat <<'EOF' | oc apply -f -
    apiVersion: v1
    kind: Namespace
    metadata:
      name: egressip-udn-repro
      labels:
        k8s.ovn.org/primary-user-defined-network: ""
        egressip-test: "true"
    EOF

    The label k8s.ovn.org/primary-user-defined-network: “” signals that workloads in the namespace should attach to a primary UserDefinedNetwork.

    The additional label egressip-test: “true” is later used by the EgressIP namespaceSelector.

    Create a Layer 2 Primary UserDefinedNetwork

    Create a primary Layer 2 UDN:

    cat <<'EOF' | oc apply -f -
    apiVersion: k8s.ovn.org/v1
    kind: UserDefinedNetwork
    metadata:
      name: repro-udn-l2
      namespace: egressip-udn-repro
    spec:
      topology: Layer2
      layer2:
        role: Primary
        subnets:
          - "10.100.0.0/24"
    EOF

    Wait for the network to become ready:

    oc wait userdefinednetwork repro-udn-l2 -n egressip-udn-repro \
      --for condition=NetworkReady=True --timeout=60s

    💡 Ensure the UDN subnet does not overlap with the cluster pod network, service network, or OVN join subnets. In this example, 10.100.0.0/24 is used as the UDN subnet.

    Enable EgressIP Assignment on a Worker Node

    Label a worker node so OVN-Kubernetes can host EgressIP assignments:

    oc label node worker-0 k8s.ovn.org/egress-assignable=””

    You can verify the label using:

    oc get nodes –show-labels

    Create the EgressIP Resource

    cat <<'EOF' | oc apply -f -
    apiVersion: k8s.ovn.org/v1
    kind: EgressIP
    metadata:
      name: repro-egressip
    spec:
      egressIPs:
        - "EGRESS_IP_ADDRESS"
      namespaceSelector:
        matchLabels:
          egressip-test: "true"
    EOF

    In a lab environment, an unused address was selected from the worker node network and associated with the node through an additional interface configured in PowerVC – and EGRESS_IP_ADDRESS was updated to match the unused address that was routable on the network.

    Verify EgressIP Assignment

    Confirm that OVN successfully assigned the EgressIP:

    oc get egressip repro-egressip -o yaml

    Review the status section and verify the address is assigned to the intended worker node.

    Example:

    status:
           items:
           – egressIP: 192.168.1.50
             node: worker-0

    Deploy a Test Pod

    Create a simple UBI-based pod:

    cat <<'EOF' | oc apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      name: egress-test-pod
      namespace: egressip-udn-repro
      labels:
        app: egress-test
    spec:
      containers:
        - name: curl
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command: ["sleep", "infinity"]
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
    EOF

    Wait until the pod becomes ready:

    oc wait pod egress-test-pod -n egressip-udn-repro --for condition=Ready --timeout=120s

    Install curl

    Access the pod:

    oc rsh -n egressip-udn-repro egress-test-pod

    Install curl:

    microdnf –setopt=cachedir=/tmp/dnf-cache –setopt=keepcache=0 install -y curl

    Validate External Connectivity

    Generate outbound traffic:

    curl -sv https://google.com/ –connect-timeout 10

    You can replace the endpoint with any reachable external service that allows source IP verification.

    Inspect Routing Inside the Pod

    Review the routing table:

    cat /proc/net/route

    Example output:

    Iface      Destination Gateway
         ovn-udn1   00000000    0100640A
         ovn-udn1   0000640A    00000000
         eth0       0000800A    0102800A

    Decoded, the routes appear as:

    DestinationGatewayInterface
    0.0.0.010.100.0.1ovn-udn1
    10.100.0.0/24Directovn-udn1
    10.128.0.0/2210.128.2.1eth0
    100.64.0.0/1610.128.2.1eth0
    100.65.0.0/1610.100.0.1ovn-udn1
    172.30.0.0/1610.100.0.1ovn-udn1

    The key observation is that the default route points to ovn-udn1, confirming that the primary UDN is acting as the pod’s primary network.

    Verify Traffic Flow

    Examine interface statistics:

    cat /proc/net/dev

    Example:

         Inter-| Receive | Transmit
         …
         eth0:      446      5
         ovn-udn1: 21828173 4698

    The significantly higher packet and byte counts on ovn-udn1 indicate that application traffic is flowing through the UserDefinedNetwork rather than the cluster default interface.

    This validates that:

    1. The pod is attached to the primary UDN.
    2. The default route is installed through ovn-udn1.
    3. Outbound traffic follows the UDN path.
    4. EgressIP can be applied to workloads selected through the namespace selector.

    Conclusion

    OpenShift’s UserDefinedNetwork capability provides a powerful way to create dedicated networking domains for applications while still taking advantage of platform services such as EgressIP. On IBM Power deployments, including PowerVM-based environments, this enables architects to combine network isolation with consistent outbound IP presentation.

    In this validation, a primary Layer 2 UDN successfully became the pod’s default network, routes were installed through ovn-udn1, and traffic counters confirmed that outbound connections traversed the UDN. Combined with EgressIP assignment, this offers a flexible pattern for workloads that require network segmentation and predictable egress behavior on OpenShift running on IBM Power.

  • 🔒 Build a private OpenShift Installer Provisioned Infrastructure on IBM PowerVS

    Ashwin Hendre’s latest guide walks through the requirements, networking considerations, IBM Cloud configuration, DNS setup, credential management, and installation workflow needed to build a private OpenShift Installer Provisioned Infrastructure on IBM PowerVS while maintaining outbound internet connectivity.

    • ✅ Private cluster architecture
    • ✅ IBM Cloud VPC and DNS configuration
    • ✅ PowerVS IPI deployment prerequisites
    • ✅ OpenShift 4.22 installation guidance
    • ✅ Cloud Credential Operator (CCO) setup

    see https://community.ibm.com/community/user/blogs/ashwin-hendre/2026/08/06/private-ocp-powervs-ipi

  • Accessing Bootstrap Logs Through VPC SSH: Complete Guide

    Based on Natalia Jordan’s IBM Community article covering IPI PowerVS bootstrap node access, VPC networking concepts, jump host setup, SSH key configuration, and bootstrap log collection for OpenShift IPI troubleshooting

    You can view the article at https://community.ibm.com/community/user/blogs/natalia-jordan/2026/02/11/accessing-bootstrap-logs-through-vpc-ssh

  • IBM Power HMC Provider 1.0 is live

    Infrastructure as Code for IBM Power systems just got easier. The IBM Power HMC Provider 1.0 is now available on the Terraform Registry, enabling administrators and automation engineers to manage IBM Power environments through Hardware Management Console (HMC) using familiar Terraform workflows.

    With the provider, you can define and manage Power resources declaratively, integrate with CI/CD pipelines, and bring consistency to infrastructure provisioning across Power estates.

    terraform {
      required_providers {
        powerhmc = {
          source  = "terraform-ibm/powerhmc"
          version = "1.0.0"
        }
      }
    }
    
    provider "powerhmc" {
      host     = var.hmc_host
      username = var.hmc_user
      password = var.hmc_password
    }
    

    Explore the provider: https://registry.terraform.io/providers/terraform-ibm/powerhmc/

  • Deploying Apache Cassandra on IBM PowerVS with Terraform and RHEL 10

    IBM Power Systems Virtual Server (PowerVS) is a great fit for data-intensive workloads. In this post, we’ll walk through deploying a single-node Apache Cassandra 5.0 instance on PowerVS using a RHEL 10 Bring-Your-Own-License (BYOL) image, fully automated with Terraform and a cloud-init shell script.

    By the end you’ll have:

    • A running PowerVS LPAR (s1022) on RHEL 10
    • IBM Semeru Runtime (OpenJ9) for Java on ppc64le
    • Apache Cassandra 5.0 installed and ready to start
    • A public network interface for SSH access

    Architecture Overview

    The deployment is split into two layers:

    Infrastructure (Terraform)

    ResourcePurpose
    ibm_pi_networkCreates a public pub-vlan network for external access
    ibm_pi_instanceProvisions the LPAR using the RHEL 10 BYOL stock image
    null_resourceSSHes into the instance after boot and runs the init script

    Configuration (init.sh)

    Once the LPAR is up, Terraform copies a templated init.sh script to the instance and executes it over SSH. The script:

    1. Registers the OS with Red Hat Subscription Manager (RHSM)
    2. Enables the RHEL 10 ppc64le base and appstream repos
    3. Updates the OS and installs tooling (wgetcurljq, etc.)
    4. Downloads and installs IBM Semeru Java 17 LTS for ppc64le from GitHub Releases
    5. Registers the cassandra YUM repo and installs Apache Cassandra 5.0
    6. Drops a cassandra-manager.service systemd unit (installed but not auto-started — you control when it runs)

    Prerequisites

    Before you begin, make sure you have:

    • An IBM Cloud account with PowerVS permissions
    • pre-provisioned PowerVS workspace (get the workspace GUID from the UI)
    • PowerVS SSH key registered in the workspace
    • Red Hat credentials (RHSM username and password) for a RHEL subscription
    • Terraform ≥ 1.3 installed locally
    • The IBM Cloud CLI with the PowerVS plugin (ibmcloud plugin install power-iaas)

    RHEL 10 BYOL image: This example uses the stock catalog image ID 585ca713-f303-45f0-a836-e9dd4f4c8f3b (RHEL10-BYOL) which is already available in PowerVS — no COS bucket or manual image upload required.


    Repository Layout

    .
    ├── main.tf                  # Provider, network, instance, and provisioner
    ├── variables.tf             # Input variable declarations
    ├── outputs.tf               # Instance ID and IP outputs
    ├── init.sh                  # cloud-init / SSH-executed bootstrap script
    └── terraform.tfvars.example # Template — copy to terraform.tfvars
    

    Step 1 — Configure Your Variables

    Copy the example variables file and fill in your values:

    cp terraform.tfvars.example terraform.tfvars
    

    Then edit terraform.tfvars:

    ibmcloud_api_key      = "YOUR_IBM_CLOUD_API_KEY"
    workspace_id          = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    region                = "us-south"
    zone                  = "dal12"
    ssh_key_name          = "my-powervs-ssh-key"
    ssh_private_key_path  = "~/.ssh/id_rsa"
    memory                = 16
    processors            = 2
    rhsm_username         = "your-redhat-username"
    rhsm_password         = "your-redhat-password"
    

    Never commit terraform.tfvars to source control — it contains your IBM Cloud API key and RHSM credentials. The .gitignore in this repo already excludes it, and all three sensitive variables are declared with sensitive = true in variables.tf so Terraform redacts them from plan/apply output.

    Variable Reference

    VariableDescriptionDefault
    ibmcloud_api_keyIBM Cloud API key
    workspace_idPowerVS workspace GUID
    regionIBM Cloud region (e.g. us-south)
    zoneIBM Cloud zone (e.g. dal12)
    ssh_key_nameSSH key registered in the PowerVS workspace
    ssh_private_key_pathLocal path to the matching private key~/.ssh/id_rsa
    memoryInstance memory in GiB16
    processorsNumber of virtual processors2
    rhsm_usernameRed Hat Subscription Manager username
    rhsm_passwordRed Hat Subscription Manager password

    Step 2 — Understand the Terraform Configuration

    Provider

    main.tf pins the IBM Cloud provider at ~> 2.4.0 and the HashiCorp null provider for the remote-exec provisioner:

    terraform {
      required_providers {
        ibm = {
          source  = "IBM-Cloud/ibm"
          version = "~> 2.4.0"
        }
        null = {
          source  = "hashicorp/null"
          version = "~> 3.0"
        }
      }
    }
    
    provider "ibm" {
      ibmcloud_api_key = var.ibmcloud_api_key
      region           = var.region
      zone             = var.zone
    }
    

    Public Network

    pub-vlan network is created to give the instance an external IP for SSH access:

    resource "ibm_pi_network" "public_net" {
      pi_cloud_instance_id = var.workspace_id
      pi_network_name      = "cassandra-public-net"
      pi_network_type      = "pub-vlan"
    }
    

    PowerVS Instance

    The LPAR is an s1022 system (Power10) using a shared processor, the stock RHEL 10 BYOL image, and attached to the public network:

    resource "ibm_pi_instance" "cassandra_node" {
      pi_cloud_instance_id = var.workspace_id
      pi_memory            = var.memory
      pi_processors        = var.processors
      pi_instance_name     = "cassandra-rhel10"
      pi_proc_type         = "shared"
      pi_sys_type          = "s1022"
      pi_image_id          = "585ca713-f303-45f0-a836-e9dd4f4c8f3b"
      pi_key_pair_name     = var.ssh_key_name
      pi_health_status     = "WARNING"
    
      pi_network {
        network_id = ibm_pi_network.public_net.network_id
      }
    
      depends_on = [ibm_pi_network.public_net]
    }
    

    pi_health_status = "WARNING" lets Terraform proceed even while the instance is still booting, which is normal — the null_resource provisioner handles the wait via SSH.

    Remote Provisioner

    After the instance is up, Terraform templates init.sh with your RHSM credentials and copies it over SSH, then executes it:

    resource "null_resource" "cassandra_init" {
      triggers = {
        instance_id = ibm_pi_instance.cassandra_node.instance_id
        script_hash = filemd5("${path.module}/init.sh")
      }
    
      connection {
        type    = "ssh"
        user    = "root"
        host    = ibm_pi_instance.cassandra_node.pi_network[0].external_ip
        agent   = true
        timeout = "10m"
      }
    
      provisioner "file" {
        content = templatefile("${path.module}/init.sh", {
          rhsm_username = var.rhsm_username
          rhsm_password = var.rhsm_password
        })
        destination = "/root/init.sh"
      }
    
      provisioner "remote-exec" {
        inline = [
          "chmod +x /root/init.sh",
          "bash /root/init.sh",
        ]
      }
    }
    

    The script_hash trigger means Terraform will re-run the provisioner if you change init.sh, which is handy during development.


    Step 3 — The Bootstrap Script (init.sh)

    init.sh is a Bash script executed on the instance as root. Here’s what it does, step by step.

    RHSM Registration

    subscription-manager register \
      --username="${rhsm_username}" \
      --password="${rhsm_password}" \
      --force
    
    subscription-manager repos \
      --enable=rhel-10-for-ppc64le-baseos-rpms \
      --enable=rhel-10-for-ppc64le-appstream-rpms
    

    The --force flag handles re-registration gracefully if the instance was previously registered. The correct repo slugs for RHEL 10 on Power LE (ppc64le) are rhel-10-for-ppc64le-*.

    OS Update and Tooling

    dnf update -y
    dnf install -y wget tar gzip jq curl
    

    IBM Semeru Java 17 (ppc64le)

    Cassandra requires Java, and IBM Semeru Runtime (OpenJ9) is the recommended JVM for Power. The script fetches the latest release URL dynamically from the GitHub API:

    SEMERU_LATEST_URL=$(curl -s https://api.github.com/repos/ibmruntimes/semeru17-binaries/releases/latest \
      | jq -r '.assets[] | select(.name | contains("jdk_ppc64le_linux")) | select(.name | endswith(".tar.gz")) | .browser_download_url')
    
    wget -qO /tmp/semeru.tar.gz "$SEMERU_LATEST_URL"
    mkdir -p /opt/ibm/semeru
    tar -xzf /tmp/semeru.tar.gz -C /opt/ibm/semeru --strip-components=1
    rm -f /tmp/semeru.tar.gz
    
    alternatives --install /usr/bin/java java /opt/ibm/semeru/bin/java 1
    echo "export JAVA_HOME=/opt/ibm/semeru" > /etc/profile.d/semeru.sh
    export JAVA_HOME=/opt/ibm/semeru
    export PATH="$JAVA_HOME/bin:$PATH"
    

    Installing under /opt/ibm/semeru keeps it clean and separate from the system Java. The alternatives registration makes java available system-wide.

    Apache Cassandra 5.0

    The official Apache Cassandra RPM repo is added and the package installed. --skip-broken is used because Cassandra’s bundled JVM dependency is skipped in favour of Semeru:

    rpm --import https://downloads.apache.org/cassandra/KEYS
    
    cat <<'EOF' > /etc/yum.repos.d/cassandra.repo
    [cassandra]
    name=Apache Cassandra 
    baseurl=https://redhat.cassandra.apache.org/50x/ 
    gpgcheck=0 
    repo_gpgcheck=0 
    gpgkey=https://downloads.apache.org/cassandra/KEYS 
    EOF
    
     dnf clean all && dnf makecache dnf install -y cassandra --skip-broken

    Systemd Service Unit

    Rather than auto-starting Cassandra, the script drops a custom cassandra-manager.service unit that explicitly sets JAVA_HOME to the Semeru installation. This prevents Cassandra from picking up an incorrect JVM:

    cat <<'EOF' > /etc/systemd/system/cassandra-manager.service
    [Unit]
    Description=Apache Cassandra Lifecycle Manager
    After=network-online.target
    Wants=network-online.target
    
    [Service]
    Type=forking
    User=cassandra
    Group=cassandra
    Environment="JAVA_HOME=/opt/ibm/semeru"
    Environment="PATH=/opt/ibm/semeru/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
    ExecStart=/usr/sbin/cassandra -p /var/run/cassandra/cassandra.pid
    PIDFile=/var/run/cassandra/cassandra.pid
    RuntimeDirectory=cassandra
    TimeoutStartSec=120
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target
    EOF
    
    systemctl daemon-reload
    

    The service is not enabled or auto-started — you decide when Cassandra runs.


    Step 4 — Deploy

    Initialize

    terraform init
    

    This downloads the IBM Cloud and null providers.

    Plan

    terraform plan -var-file="terraform.tfvars"
    

    Review the output — you should see three resources being created: ibm_pi_network.public_netibm_pi_instance.cassandra_node, and null_resource.cassandra_init.

    Apply

    terraform apply -var-file="terraform.tfvars"
    

    Total deployment time is approximately 10–15 minutes:

    • Network creation: ~1 minute
    • Instance boot: ~5 minutes
    • init.sh execution (OS update + Java + Cassandra): ~5–10 minutes

    Step 5 — Verify

    Once terraform apply completes, grab the IP from the outputs:

    terraform output instance_external_ip_address
    

    SSH in as root:

    ssh root@<external_ip>
    

    Confirm IBM Semeru is the active JVM:

    java -version
    # Expected output:
    # openjdk version "17.x.x" ...
    # IBM Semeru Runtime Open Edition ...
    # Eclipse OpenJ9 ...
    

    Start Cassandra:

    systemctl start cassandra-manager
    systemctl status cassandra-manager
    

    Once Cassandra is up (give it 30–60 seconds), verify the node is healthy:

    nodetool status
    

    You should see a single node listed as UN (Up, Normal):

    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Tokens  Owns   Host ID   Rack
    UN  127.0.0.1  ...        16      100.0% <uuid>    rack1
    

    To persist Cassandra across reboots:

    systemctl enable cassandra-manager
    

    Outputs Reference

    OutputDescription
    instance_idPowerVS LPAR instance ID
    instance_ip_addressPrivate IP on the attached network
    instance_external_ip_addressPublic/external IP (used for SSH)

    Security Considerations

    • Credentials in terraform.tfvars — never commit this file. It is already listed in .gitignore.
    • Sensitive variables — ibmcloud_api_keyrhsm_username, and rhsm_password are marked sensitive = true in variables.tf, so Terraform will not print them in plan or apply output.
    • SSH agent forwarding — the provisioner connection uses agent = true, which relies on your local SSH agent. Make sure the private key matching ssh_key_name is loaded (ssh-add ~/.ssh/id_rsa).
    • Firewall — the pub-vlan network exposes the instance publicly. Consider restricting inbound access to port 22 (SSH) and Cassandra’s native transport port 9042 to trusted CIDRs via IBM Cloud security groups.

    Cleaning Up

    To destroy all resources created by this deployment:

    terraform destroy -var-file="terraform.tfvars"
    

    This will remove the PowerVS instance and the public network. The RHEL 10 stock image is not deleted (it is a shared catalog image, not a user-imported resource).


    Summary

    This walkthrough showed how to:

    1. Use Terraform with the IBM-Cloud/ibm provider to stand up a PowerVS LPAR on RHEL 10
    2. Bootstrap the instance via SSH using a templated shell script
    3. Install IBM Semeru Runtime (OpenJ9) for ppc64le — the right JVM for Power
    4. Add the official Apache Cassandra 5.0 RPM repo and install Cassandra
    5. Register a custom systemd service that pins JAVA_HOME to the Semeru installation

    The full source is available in this repository — clone it, fill in your terraform.tfvars, and you’ll have Cassandra running on PowerVS in under 15 minutes.

    Note the source is attached

  • Enabling CPU Manager and CPU Pinning in OpenShift

    By default, OpenShift is a greedy scheduler, when a Pod starts it grabs the first requested resource CPU/Memory. There is a reserved bit of memory and CPU for kubelet and System services saved. A Pod may end up spanning different NUMA domains

    The CPU Manager packs each Pod onto NUMA domains:.

    1. single-numa-node a pod might fail to schedule on a node that has enough total CPU capacity but lacks enough contiguous cores. If you go to schedule a 8 vCPU pod, and you have 4vcpu on one node, and 4 vcpu on another, it won’t be able to allocate ending with a TopologyAffinityError.
    2. restricted packs it into as few NUMA nodes as possible. This one is probably preferred as it allows for rounding.

    Keep in mind, this applies for the whole system, and you may want to isolate to only a few workers you can put them in a separate pool. You can see https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/scalability_and_performance/using-cpu-manager#setting_up_cpu_manager_using-cpu-manager-and-topology-manager

    apiVersion: machineconfiguration.openshift.io/v1
    kind: KubeletConfig
    metadata:
      name: cpumanager-enabled
    spec:
      machineConfigPoolSelector:
        matchLabels:
          pools.operator.machineconfiguration.openshift.io/worker: ""
      kubeletConfig:
        cpuManagerPolicy: static 
        cpuManagerReconcilePeriod: 5s
        topologyManagerPolicy: single-numa-node 
        reservedSystemCPUs: "0,1" 
        memoryManagerPolicy: Static
    

    If a Pod is idle, those cores sit idle. They won’t share the spare cycles of another Pod.

    If a Pod used 100m, it’ll need to specificy round numbers now cpu: "2" in some cases.

    For large workloads , CPU pinning is most effective when the pod is sized to fit within a single NUMA node. If your pod is so large that it spans multiple sockets, the benefits of pinning diminish unless the application itself is NUMA-aware.

    You can see more details at OpenShift 4.20: Using CPU Manager and Topology Manager [docs.redhat.com]

  • Workloads: From Bare Metal to KubeVirt

    Workloads run on full physical servers or are managed as virtual machines. KubeVirt brings cloud-native approach to fleet management and makes VMs first-class Kubernetes resources.

    1. Physical Infrastructure

    Typically, things begin with a physical server. A Baseboard Management Controller (BMC) provides hardware management, while firmware initializes CPU and memory resources. At this stage, all resources belong to a single system with no virtualization.

    Key point: Physical hardware only, with no abstraction layer.


    2. Traditional Virtualization

    A host operating system such as RHEL, SUSE, or Ubuntu runs on the server. Libvirt and QEMU provide virtualization services, allowing multiple VMs to share the same hardware.

    Key point: Virtual machines improve hardware utilization while remaining tied to individual hosts.


    3. Kubernetes + KubeVirt

    KubeVirt integrates virtualization into Kubernetes. Instead of managing VMs directly on each host, Kubernetes manages VM lifecycle through KubeVirt components such as Virt Operator, Virt Controller, and Virt Launcher.

    Virtual machines become Kubernetes-managed workloads that can be scheduled, automated, and operated alongside containers.


    Conclusion

    The progression is straightforward Physical Server run Libvirt/QEMU Virtualization and are managed as a fleet with Kubernetes-Orchestrated Virtualization with KubeVirt.

    Note: based on an ad-hoc presentation….

  • Aside: cert-manager: If You’re Running On-Prem Without Direct DNS Access

    One of the more common challenges in disconnected or tightly controlled on-prem OpenShift environments is certificate validation when the cluster doesn’t have direct access to external DNS resolvers.

    If cert-manager’s DNS-01 validation cannot reach the appropriate DNS infrastructure, certificate issuance may fail even when the DNS records themselves are correct. In these environments, you can configure the cert-manager operator to use specific internal recursive DNS servers instead of relying on default resolver behavior.

    The following patch updates the cert-manager configuration to query only the designated recursive nameservers:

    oc patch certmanager cluster --type=merge -p='{
      "spec": {
        "controllerConfig": {
          "overrideArgs": [
            "--dns01-recursive-nameservers=10.0.10.4:53,10.0.10.5:53",
            "--dns01-recursive-nameservers-only"
          ]
        }
      }
    }'
    

    This approach is particularly useful in air-gapped, restricted-network, or enterprise environments where outbound DNS traffic is blocked and all name resolution must flow through approved internal resolvers.

    A small change, but often the difference between endless certificate troubleshooting and a clean, automated issuance process.

  • Capturing Kernel Crash Dumps on Persistent Memory on OpenShift Container Platform on IBM Power

    Co-authored and reposted here…

    Kaushik Talathi, Engineer, OpenShift on IBM Power

    Paul Bastide, Engineering Lead, OpenShift on IBM Power

    When a worker node panics, the kernel’s last words are the most valuable debugging artifact you’ll ever get — if you managed to capture them. On your OpenShift Container Platform on IBM Power or on IBM Power Virtual Server, you can capture kernel crash dumps to available persistent memory (pmem) on the nodes. kdump is able to write vmcores to a pmem-backed filesystem: it’s fast, it’s dedicated, and it keeps multi-gigabyte crash dumps off the root disk or NFS servers.

    Using this document, you learn how to: create a pmem namespace with ndctl, format and mount it via MachineConfig systemd units, configure kdump through butane, trigger a crash with echo c > /proc/sysrq-trigger — and the vmcore landed exactly where it should, on /var/mnt/pmem.

    Red Hat CoreOS is an immutable OS which doesn’t shift with ndctl; you can’t just use it by default. This document shows how to use a privileged pod with access to /dev and /sys, running ndctl create-namespace --mode=fsdax against the host’s NVDIMM subsystem. This day-2 action is useful when troubleshooting and not intended to be used by default – in exceptional debugging of storage problems.

    When we don’t have direct access to tools on the file system, we use oneshot systemd unit, delivered by MachineConfig, that runs the privileged container to execute:

    [Unit]
    Description=Create PMEM namespace (ndctl via privileged container)
    ConditionPathExistsGlob=/sys/bus/nd/devices/region*
    ConditionPathExists=!/dev/pmem0
    Wants=network-online.target
    After=network-online.target
    Before=format-pmem0.service var-mnt-pmem.mount
    
    [Service]
    Type=oneshot
    RemainAfterExit=yes
    TimeoutStartSec=600
    ExecStart=/usr/local/bin/pmem-namespace.sh
    
    [Install]
    WantedBy=multi-user.target
    

    The two Condition lines are doing the heavy lifting. If /dev/pmem0 already exists, there’s nothing to do. If the node has no NVDIMM regions at all, there’s nothing that can be done. In both cases systemd skips the unit — skipped, not failed — and boot proceeds normally. That’s the behavior we want across a fleet where not every worker has pmem.

    The script it calls is small and idempotent:

    #!/usr/bin/env bash
    set -euo pipefail
    
    NDCTL_IMAGE="${NDCTL_IMAGE:-quay.io/centos/centos:stream10}"
    AUTHFILE="/var/lib/kubelet/config.json"
    
    [ -b /dev/pmem0 ] && { echo "already present"; exit 0; }
    ls /sys/bus/nd/devices/region* >/dev/null 2>&1 || { echo "no NVDIMM regions"; exit 0; }
    
    AUTH_ARGS=()
    [ -f "${AUTHFILE}" ] && AUTH_ARGS=(--authfile "${AUTHFILE}")
    
    podman run --rm --privileged "${AUTH_ARGS[@]}" \
      -v /dev:/dev -v /sys:/sys \
      "${NDCTL_IMAGE}" \
      bash -c 'command -v ndctl >/dev/null 2>&1 || dnf -y -q install ndctl; ndctl create-namespace --mode=fsdax'
    
    udevadm settle
    [ -b /dev/pmem0 ] || { echo "expected /dev/pmem0" >&2; exit 1; }
    

    By default it pulls CentOS Stream 10 and installs ndctl on the fly. For disconnected environments or faster boots, prebuild a two-line image (FROM quay.io/centos/centos:stream10 + dnf -y install ndctl daxctl), push it to your registry, and point NDCTL_IMAGE at it — the script already passes the node’s pull secret via --authfile.

    The startup flows:

    pmem-namespace.service  →  format-pmem0.service  →  var-mnt-pmem.mount  →  kdump.service
       (create if needed)        (mkfs once, ever)         (xfs, nofail)        (arm capture kernel)
    

    All of it — the script, the units, the crashkernel= kernel arguments, /etc/kdump.conf, and /etc/sysconfig/kdump — lives in one butane file rendered to a single MachineConfig. The full listing is in the companion setup document; render and apply is the usual two-liner:

    butane 99-worker-pmem-kdump.bu -o 99-worker-pmem-kdump.yaml
    oc apply -f 99-worker-pmem-kdump.yaml
    

    What to do with the cores

    Dumps arrive under /var/mnt/pmem/<ip>-<date>-<time>/ as vmcorevmcore-dmesg.txt, and kexec-dmesg.log. Three habits worth adopting:

    Read vmcore-dmesg.txt first. It’s the crashed kernel’s ring buffer as plain text, panic backtrace at the bottom. A large fraction of crashes are diagnosed from this file alone, no tooling required.

    Use toolbox for deep analysis. This is where toolbox shines. From oc debug node/<node> → chroot /host → toolbox, you get a privileged support-tools container where you can dnf install crash plus the matching kernel-debuginfo and open the vmcore in place. Debuginfo is huge and must match the crashed kernel exactly, so in practice we often stream the dump off the node instead —

    oc debug node/worker-0 -- \
      tar czf - -C /host/var/mnt/pmem 127.0.0.1-2026-07-09-12:06:09 > vmcore-worker-0.tgz
    

    — and run crash on a RHEL box where the right debuginfo installs cleanly.

    Prune automatically. Persistent memory is not infinite, and a filtered vmcore is still roughly proportional to RAM. A tiny systemd timer in the same MachineConfig deletes dump directories older than 14 days, and its ConditionPathIsMountPoint=/var/mnt/pmem means it too is a no-op on nodes without pmem.

    Verifying it

    After the MachineConfigPool converges:

    oc debug node/worker-0
    chroot /host
    systemctl status kdump
    cat /sys/kernel/kexec_crash_loaded    # 1 means armed
    findmnt /var/mnt/pmem                 # xfs on /dev/pmem0
    

    And in a maintenance window, once you reboot, you can see the vmcore survived so you don’t have to wonder:

    echo c > /proc/sysrq-trigger
    # node dumps, reboots, and then:
    find /var/mnt/pmem -type f
    /var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore
    /var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/vmcore-dmesg.txt
    /var/mnt/pmem/127.0.0.1-2026-07-09-12:06:09/kexec-dmesg.log
    

    One caveat for mixed fleets: on a node where the pmem mount never exists, kdump’s path /var/mnt/pmem falls back to the root filesystem. If only some of your workers carry pmem, put this MachineConfig on a dedicated pool (e.g., a worker-pmem role), or switch kdump.conf to a direct dump target (xfs /dev/pmem0 + path /) so kdump fails loudly rather than dumping to /.

    Wrapping up

    The pattern here generalizes beyond pmem: when RHCOS doesn’t have the tool, a privileged container run by podman from a oneshot systemd unit is how you run it at boot — with Condition* directives making the whole thing safe on nodes where it doesn’t apply. Toolbox is the interactive face of the same idea, and it earns its keep again when it’s time to open the vmcore with crash.

    Credit where it’s due: the namespace-creation discovery, the kdump-to-pmem configuration, and the end-to-end crash validation are Kaushik Talathi’s work — this post just wires his findings into the boot sequence so nobody ever has to remember the privileged pod again.

    Validated on OpenShift 4.18, RHCOS (RHEL 9.x, kernel 5.14.0-570.x).

    References

  • Aside: Applying Kernel Tunnables quickly (randomize_va_space)

    When a Tuned profile changes a sysctl such as:

    [sysctl]
    kernel.randomize_va_space=0

    You can use TuneD Custom Resource:

    apiVersion: tuned.openshift.io/v1
    kind: Tuned
    metadata:
      name: disable-aslr
      namespace: openshift-cluster-node-tuning-operator
    spec:
      profile:
      - name: disable-aslr-profile
        data: |
          [main]
          summary=Disable ASLR
          include=openshift-node
    
          [sysctl]
          kernel.randomize_va_space=0
    
      recommend:
      - priority: 20
        profile: disable-aslr-profile
        match:
        - label: node-role.kubernetes.io/worker

    The Node Tuning Operator/tuned applies it dynamically via the sysctl interface, so a reboot is typically not required. The new value takes effect immediately on the node after the profile is applied.

    You can verify:

    oc debug node/<node>
    chroot /host
    
    sysctl kernel.randomize_va_space

    or

    cat /proc/sys/kernel/randomize_va_space

    Disabling ASLR through a Tuned object, the setting is applied without a node reboot. You can usually observe the change within seconds after the profile becomes active.