Blog

At DevConf.CZ 2025, a standout session from Alessandro Di Stefano and Prashanth Sundararaman introduced the Outrigger project, a forward-thinking initiative aimed at transforming Kubernetes scheduling into a dynamic, collaborative ecosystem. Building on the success of the Multiarch Tuning Operator for OpenShift, Outrigger leverages Kubernetes’ scheduling gates to go beyond traditional multi-architecture scheduling.

👉 Watch the full session here:

Excellent work by that team.

Cross post from https://community.ibm.com/community/user/blogs/paul-bastide/2025/06/09/using-nx-gzip-in-your-red-hat-openshift-container

The Power10 processor features an on-chip accelerator that is called the nest accelerator unit (NX unit). The coprocessor features that are available on the Power10 processor are similar to the features of the Power9 processor. These coprocessors provide specialized functions, such as the Industry-standard Gzip compression and decompression, Random number generation and AES and Secure Hash Algorithm (SHA) cryptography.

This article outlines how to use nx-gzip in a non-privileged container in Red Hat OpenShift Container Platform on IBM Power. You must have deployed a cluster with workers with a processor compatibility of IBM Power 10 or higher. The Active Memory Expansion feature must be licensed.

Build the power-gzip selftest binary

The test binary is used to show the feature is working and you can use the selftest and sample code to integrate in your environment.

1. Login to the PowerVM instance running Red Hat Enterprise Linux 9
2. Install required build binaries

dnf install make git gcc zlib-devel vim util-linux-2.37.4-11.el9.ppc64le -y

3. Setup the Clone repository

git clone https://github.com/libnxz/power-gzip
cd power-gzip/

4. Run the tests

./configure 
cd selftests
make

5. Find the created test files

# ls g*test -al
-rwxr-xr-x. 1 root root 74992 Jun  9 08:24 gunz_test
-rwxr-xr-x. 1 root root 74888 Jun  9 08:24 gzfht_test

You are ready to test it.

Setup the NX-GZip test deployment

Download the examples repository and setup kustomization, and configure cri-o so you can deploy and use /dev/crypto/nx-gzip in a container.

1. Install Kustomization tool for the deployment

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
sudo mv kustomize /usr/local/bin
kustomize -h

2. Clone the ocp4-power-workload-tools repository

git clone https://github.com/IBM/ocp4-power-workload-tools
cd ocp4-power-workload-tools

3. Configure the worker nodes to use /dev/crypto/nx-gzip as an allowed_device.

oc apply -f ocp4-power-workload-tools/manifests/nx-gzip/99-worker-crio-nx-gzip.yaml

4. Export kubeconfig using export KUBECONFIG=~/.kube/config
5. Setup the nx-gzip test Pod as below

cd manifests/nx-gzip
kustomize build . | oc apply -f - 

6. Resulting running pod as below

# oc get pod -n nx-gzip-demo
NAME               READY   STATUS    RESTARTS   AGE
nx-gzip-ds-2mlmh   1/1     Running   0          3s

You are ready to test nx-gzip.

To test with Privileged mode, you can use nx-gzip-privileged.

Copy the Test artifact into the running Pod and Run the Test Artifact

1. Copy the above created executable files to the running pod

# oc cp gzfht_test nx-gzip-ds-2mlmh:/nx-test/

2. Access the pod shell and confirm the Model name is Power10 or higher.

# oc rsh nx-gzip-ds-2mlmh
sh-5.1# lscpu | grep Model
Model name:                           POWER10 (architected), altivec supported
Model:                                2.0 (pvr 0080 0200)

3. Create a test file for testing

sh-5.1# dd if=/dev/random of=/nx-test/test bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431494 s, 243 MB/s
sh-5.1#

4. Run the tests in pod

sh-5.1# /nx-test/gzfht_test /nx-test/test
file /nx-test/test read, 1048576 bytes
compressed 1048576 to 1105994 bytes total, crc32 checksum = a094fbab
sh-5.1# echo $?
0

If it shows as compressed and the return code is 0 and as above then its considered as PASS.

You’ve seen how the nx-gzip works in Pod. You can also combine with the Node Feature Discovery to label each Node Resource with cpu-coprocessor.nx_gzip=true

Thank you for your time and good luck.

Reference

1. IBM Power10 Scale Out Servers Technical Overview S1012, S1014, S1022s, S1022 and S1024
2. Exploitation of In-Core Acceleration of POWER Processors for AIX
3. POWER NX zlib compliant library
4. Db2: Hardware accelerated backup and log file compression

If you are using the IBM Virtual Fibre Channel adapter with your OpenShift on Power installation, you can use these steps to get the log details.

Here are the steps to get the ibmvfc from the nodes which are failing:

Grabbing the ibmvfc logs

ibmvfc is the driver for the virtual fibre channel adapters.

To setup ibmvfc logging:

1. Login as a cluster-admin

# export KUBECONFIG=/root/openstack-upi/auth/kubeconfig
# oc get MachineConfigPool -o=jsonpath='{range.items[*]}{.metadata.name} {"\t"} {.status.nodeInfo.kubeletVersion}{"\n"}{end}'
master
worker

2. For each of these listed MachineConfigPools, let’s create 99-<mcp-name>-vfc.yaml. These systems will reboot.

# cat << EOF > 99-worker-vfc.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: "worker"
  name: 99-worker-vfc
spec:
  kernelArguments:
    - 'scsi_mod.scsi_logging_level=4096'
    - 'ibmvfc.debug=1'
    - 'ibmvfc.log_level=3'
EOF

# cat << EOF > 99-master-vfc.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: "master"
  name: 99-master-vfc
spec:
  kernelArguments:
    - 'scsi_mod.scsi_logging_level=4096'
    - 'ibmvfc.debug=1'
    - 'ibmvfc.log_level=3'
EOF

3. Let’s apply the yamls, one at a time:

# oc apply -f 99-worker-vfc.yaml
machineconfig.machineconfiguration.openshift.io/99-worker-vfc created

4. Wait for the MachineConfigPool to come back up, such as worker:

# oc wait mcp/worker --for condition=Ready --timeout=30m

5. Verify each Machine Config Pool is done updating:

The following shows the worker pool is updating:

# oc get mcp worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-b93fdaee39cd7d38a53382d3c259c8ae   False     True       True       2              1                   1                     1                      8d

The following shows the worker pool is Ready:

# oc get mcp worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
worker   rendered-worker-b93fdaee39cd7d38a53382d3c259c8ae   True     False       False       2              2                   0                     2                      8d

6. Spot check the updates…

a. List the nodes oc get nodes b. Connect to one of the nodes oc debug node/worker-0 c. Change context to /host chroot /host d. verify kernel argument contain the three values we set.

# rpm-ostree kargs
rw $ignition_firstboot  ostree=/ostree/boot.1/rhcos/d7d848ba24dcacb1aba663e9868d4bd131482d9b7fecfa33197f558c53ae5208/0 ignition.platform.id=powervs root=UUID=06207aa5-3386-4044-bcb6-750e509d7cf0 rw rootflags=prjquota boot=UUID=6c67b96e-4e01-4e01-b8e5-ffeb4041bee2 systemd.unified_cgroup_hierarchy=1 cgroup_no_v1="all" psi=0 scsi_mod.scsi_logging_level=4096 ibmvfc.debug=1 ibmvfc.log_level=3 rd.multipath=default root=/dev/disk/by-label/dm-mpath-root

7. Wait for the error to occur, get the console logs and the journalctl --dmesg output from the node.

You’ll end up with a bunch of messages like:

[    2.333257] ibmvfc 30000004: Partner initialization complete
[    2.333308] ibmvfc 30000004: Sent NPIV login
[    2.333336] ibmvfc: Entering ibmvfc_alloc_mem
[    2.333340] ibmvfc: Entering ibmvfc_alloc_queue
[    2.333343] ibmvfc: Entering ibmvfc_init_event_pool
[    2.333402] ibmvfc: Leaving ibmvfc_alloc_mem
[    2.333439] ibmvfc: Entering ibmvfc_init_crq
[    2.333443] ibmvfc: Entering ibmvfc_alloc_queue
[    2.333446] ibmvfc: Entering ibmvfc_init_event_pool
[    2.333482] ibmvfc: Leaving ibmvfc_init_event_pool
[    2.333743] ibmvfc: Leaving ibmvfc_init_crq

Once we’ve grabbed this level of detail, we can delete the MachineConfig and it’ll reboot and reset the kernel arguments.

And you can share the logs with support.

Please only use this under guidance.

Reference

https://www.ibm.com/docs/en/linux-on-systems?topic=commands-scsi-logging-level