Category: OpenShift

  • Managing NVDIMM Devices with ndctl on OpenShift 4: A Complete Setup Guide

    Managing Non-Volatile Memory devices (like /dev/nmem0) using ndctl from within a container on an OpenShift 4 (RHCOS) cluster presents a unique challenge. Because ndctl interacts directly with the kernel’s NVDIMM Firmware Interface Table (NFIT) and requires raw ioctl access to the devices, standard container sandboxing will block it. Furthermore, Red Hat CoreOS (RHCOS) is an immutable operating system, meaning day-two hardware management requires specific architectural considerations.

    This guide breaks down the exact permissions, mounts, OpenShift constructs, and operational best practices required to configure and maintain this setup reliably.

    Bypassing the Sandbox

    To run a container that can access host hardware, you must configure high-level host privileges and explicit system mounts.

    1. OpenShift Security Context Constraints (SCC) OpenShift enforces strict security by default. The Pod’s ServiceAccount must be granted the privileged SCC. This automatically transitions the pod’s SELinux context to spc_t (Super Privileged Container), which is required to bypass SELinux restrictions when accessing raw host device nodes on RHCOS.

    # Grant the privileged SCC to your ServiceAccount
    oc adm policy add-scc-to-user privileged -z my-ndctl-sa -n my-namespace
    

    2. Pod Security Context Within your Pod deployment YAML, the container must explicitly request privileged execution. This grants the CAP_SYS_ADMIN capability (essential for ndctl hardware ioctls) and disables device cgroup filtering.

    securityContext:
      privileged: true
    

    3. Required Host Mounts The ndctl utility relies heavily on sysfs to discover the NVDIMM topology and /dev to issue commands. You must mount the host’s /dev and /sys directories into the container.

    • /dev: Required to access /dev/nmem0/dev/ndctl0, etc.
    • /sys: Required because ndctl scans /sys/class/nd/ and /sys/bus/nd/ to build the device tree.

    Part 2: Deployment Configuration

    Here is a complete example of how a CentOS 10 container pod would need to be configured to run ndctl successfully on your OpenShift cluster.

    apiVersion: v1
    kind: Pod
    metadata:
      name: ndctl-manager
      namespace: my-namespace
    spec:
      serviceAccountName: my-ndctl-sa
      containers:
      - name: centos10-ndctl
        image: quay.io/centos/centos10:latest
        command: ["/bin/sleep", "infinity"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: host-dev
          mountPath: /dev
        - name: host-sys
          mountPath: /sys
      volumes:
      - name: host-dev
        hostPath:
          path: /dev
          type: Directory
      - name: host-sys
        hostPath:
          path: /sys
          type: Directory
    

    Once it starts, run dnf install -y ndctl in the container.

    Once deployed, you can execute into the container and run your ndctl commands. The OS mismatch between the RHCOS 9 host and the CentOS 10 container is fine, provided the kernel API/ABIs for NVDIMMs remain compatible. Ensure ndctl is installed in your custom image (dnf install ndctl). When running commands (e.g., ndctl create-namespace -f -e namespace0.0 -m fsdax), the utility will now have the necessary hardware visibility and CAP_SYS_ADMIN privileges.

    Operational Best Practices for RHCOS

    Keeping this setup working reliably as your cluster scales and upgrades requires adhering to OpenShift’s declarative nature.

    Managing Kernel Modules via MachineConfig The underlying RHCOS 9 host must have the correct NVDIMM kernel modules loaded to expose /dev/nmem0. OpenShift does not always load NVDIMM/PMEM modules by default. Do not rely on a manual modprobe inside your container, as it will not survive a node reboot. Instead, create a MachineConfig object to drop a configuration file into /etc/modules-load.d/ on the RHCOS nodes for modules like libnvdimmnd_pmem, and dax_pmem.

    The RHCOS Immutability Rule Hardware configurations saved to the physical NVDIMM’s label area via ndctl will survive a reboot. However, if your workflow requires changing host-level OS configurations—such as adding a udev rule for permissions or modifying /etc/fstab—do not do this directly from the container via the /host mount. Deploy host file changes via the Machine Config Operator (MCO) to prevent them from being overwritten during drift-reconciliation cycles.

    Deploying as a DaemonSet with Node Affinity Running a static Pod is a fragile way to manage hardware in a dynamic cluster. To handle node failures and scaling, use the Node Feature Discovery (NFD) Operator to automatically label nodes that physically contain NVDIMM hardware. Then, wrap your management container in a DaemonSet configured with a nodeSelector or nodeAffinity targeting those NFD labels.

    OpenShift Upgrades and ABI Compatibility You are running a CentOS 10 user-space against an RHCOS 9 kernel space. While Linux maintains strong backward compatibility for ioctl calls, there is a slight risk of divergence. During major OpenShift upgrades, test your container image in a non-production environment to verify that ndctl commands still successfully read the NFIT tables without ABI mismatch errors.

    Interaction with Storage Operators Configuring the memory is only step one. If your goal is to let Kubernetes workloads consume the NVDIMM as persistent storage, you will likely need the Local Storage Operator (LSO) or a CSI driver (like the PMEM-CSI driver) to discover the newly formatted device. Ensure your container provisions the namespaces in the exact mode (fsdaxdevdax, or sector) expected by your storage operator.

  • Custom Metrics Autoscaling with KEDA: Reliable, Observable, and Ready for IBM Power

    Custom Metrics AutoScaler, Red Hat’s KEDA build, makes Kubernetes autoscaling more flexible, event-driven, and production-ready. With support for diverse scalers, fallback behavior, lifecycle events, and observability signals, CMA helps platform teams scale workloads based on the metrics that matter most to their applications.

    A key reliability improvement is KEDA’s fallback behavior. When an external metrics source becomes unavailable, CMA can preserve the last-known replica count instead of making unsafe scaling decisions. This behavior is validated for standard Deployments and also for Argo Rollouts, helping teams maintain stability during temporary metrics outages while supporting progressive delivery patterns. You should use the GitOps Operator to use with IBM Power and OpenShift.

    KEDA also improves event-driven visibility through the CloudEventSource custom resource. Scaling lifecycle events can be emitted as structured CloudEvents, including meaningful sourcesubject, and type fields. This makes it easier to integrate KEDA activity with event routers, audit systems, and operational workflows.

    The scaler ecosystem remains broad and practical. The cron scaler supports scheduled scale-up and scale-down windows, making it useful for predictable traffic patterns. The kubernetes-workload scaler enables one workload to scale based on the pod count of another workload, with activationValue thresholds helping avoid unnecessary scaling from low-signal activity.

    Resource-based autoscaling is also covered through CPU and memory scalers, which scale deployments when container utilization crosses configured percentage thresholds. These provide familiar autoscaling behavior while remaining part of KEDA’s unified scaling model.

    For event streaming workloads, the Kafka scaler supports scenarios such as consumer group lag, offset commit policies, and partition distribution strategies. This makes KEDA a strong fit for streaming systems where scale should reflect backlog and consumption pressure.

    For metrics-driven platforms, the Prometheus scaler allows deployments to scale from the result of a configurable PromQL query. This gives teams the freedom to scale from application, infrastructure, or business metrics already exposed through Prometheus.

    KEDA also strengthens operational confidence through observability. Tests validate that KEDA exports correct OpenTelemetry metrics to collectors and exposes well-formed Prometheus metrics from operator and adapter endpoints. These signals help teams monitor scaler health, adapter behavior, reconciliation activity, and autoscaling outcomes.

    Finally, Custom Metrics AutoScaler v2.19 adds support for IBM Power, expanding deployment options for organizations running Kubernetes on Power-based infrastructure.

    Together, these features make CMA a powerful Custom Metrics AutoScaler for modern Kubernetes environments: flexible in what it scales from, resilient when dependencies fail, observable by default, and increasingly portable across infrastructure platforms.

    Go forth and build with IBM Power and CMA. https://community.ibm.com/community/user/blogs/paul-bastide/2026/06/17/custom-metrics-autoscaler-operator-v2191-supports

  • Custom Metrics Autoscaler Operator v2.19.1 supports IBM Power

    Today, Red Hat released v2.19.1 of Custom Metrics AutoScaler. Based on KEDA, the optional operator increases or decreases the number of Pods, custom resource, or Job based on custom metrics; it’s not only about CPU and memory. You can control your applications to control usage.

    To get started:

    1. On OpenShift Container Platform web console, click Ecosystem > Installed Operators.
    2. Select Custom Metrics Autoscaler.
    3. On the Operator Details page, click the KedaController tab.
    4. On the KedaController tab, click Create KedaController and edit the file.
    apiVersion: keda.sh/v1alpha1
    kind: KedaController
    metadata:
      name: keda
      namespace: openshift-keda
    spec:
      metricsServer:
        logLevel: "0"
      operator:
        logEncoder: console
        logLevel: info
      serviceAccount: {}
      watchNamespace: ""
    

    You can integrate with Kafka

    helm repo add strimzi https://strimzi.io/charts/
    helm repo update
    helm upgrade --install --namespace strimzi --wait strimzi strimzi/strimzi-kafka-operator --version 0.47.0 --set watchAnyNamespace=true
    

    Good luck…

    References

    1. Red Hat OpenShift Docs: Chapter 3. Automatically scaling pods with the Custom Metrics Autoscaler Operator
    2. Red Hat Developer: Custom Metrics Autoscaler on OpenShift
    3. keda.sh: Kubernetes Event-driven Autoscaling
    4. Custom Metrics Autoscaler Operator v2.19.1 supports IBM Power
  • 2026-06: Additional IBM Power Open Source Images on the IBM Container Registry

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). These images are available for no-charge and can be used in your development and production environments.

    Image NameTag NameProject LicensesImage Pull CommandLast Published On
    kafka4.1.0-bvApache-2.0podman pull icr.io/ppc64le-oss/kafka-ppc64le:4.1.0-bvJune 10, 2026
    ansible-acme-test-container2.3.0GPL-3.0, Apache License 2.0podman pull icr.io/ppc64le-oss/ansible-acme-test-container-ppc64le:2.3.0June 10, 2026
    vllm0.9.1Apache-2.0podman pull icr.io/ppc64le-oss/vllm-ppc64le:0.9.1June 9, 2026

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/

  • How to Run Multiple Clusters with one bastion node

    To run multiple OpenShift clusters from one bastion requires managing dhcpd, named, http, haproxy with isolated configurations.

    After deploying with ocp4-upi-powervm, you can ‘move’ the configuration over

    1. dhcpd enables booting the rhcos nodes, which then can grab their configuration. dhcpd support include statements, allowing you to split subnets, host reservations, and cluster-specific configurations into separate files.

      1. Create the conf.d directory: mkdir -p /etc/dhcp/conf.d
      2. Modify your main /etc/dhcp/dhcpd.conf to include the directory. Add this at the bottom of the file: include "/etc/dhcp/conf.d/ocp-cluster-1.conf";
      3. Create the file /etc/dhcp/conf.d/ocp-cluster-1.conf – you’ll have to give the host unique names.
      subnet 10.20.176.0 netmask 255.255.240.0 {
      interface eth0;
          # Static entries
          host bootstrap { hardware ethernet fa:16:3e:ff:b7:b2; fixed-address 10.20.188.84; }
          host master-0 { hardware ethernet fa:16:3e:9b:c5:89; fixed-address 10.20.188.206; }
          host master-1 { hardware ethernet fa:16:3e:b7:ba:16; fixed-address 10.20.188.62; }
          host master-2 { hardware ethernet fa:16:3e:14:2c:ff; fixed-address 10.20.188.166; }
          host worker-0 { hardware ethernet fa:16:3e:97:7b:1b; fixed-address 10.20.188.79; }
          host worker-1 { hardware ethernet fa:16:3e:62:39:fe; fixed-address 10.20.188.234; }
          host worker-2 { hardware ethernet fa:16:3e:23:54:0a; fixed-address 10.20.188.131; }
          # this will not give out addresses to hosts not listed above
          #deny unknown-clients;
      
          # this is PXE specific
          filename "boot/grub2/powerpc-ieee1275/core.elf";
      
          next-server 10.20.188.128;
          }
      
      1. Restart the systemd service systemctl restart dhcpd
    2. If you are hosting ignition files on httpd on port 8080.

      1. Create the ignition folder mkdir -p /var/www/html/ignition/{ocp-cluster-1,ocp-cluster-2}
      2. Copy the ignition files into /var/www/html/ignition/ocp-cluster-#
      3. Or Download the ignitions curl -k -H "Accept: application/vnd.coreos.ignition+json;version=3.4.0" -o /var/www/html/ignition/power.ign https://api-int.XYZ.powervs-openshift-ipi.cis.ibm.net:22623/config/power
      4. Restore selinux restorecon -r /var/www/html/ignition
    3. HAProxy allows us to use separate use_backend and acl

      1. Edit /etc/haproxy/haproxy.cfg
      2. Add acl for the domain name based on hostname
      frontend https-all
      mode        tcp
      option      tcplog
      
      bind        *:443
      
      acl 02-https-ci req_ssl_sni -m end .mycluster1.ibm.net
      use_backend https-workers-02 if 02-https-ci
      
      acl 03-https req_ssl_sni -m end .mycluster2.ibm.net
      use_backend https-workers-03 if 03-https
      
      1. Create a backend target for the above:
      backend https-workers-03
      mode        tcp
      balance     roundrobin
      server      master1 192.168.3.11:443 check
      server      master2 192.168.3.12:443 check
      server      master3 192.168.3.13:443 check
      server      worker1 192.168.3.51:443 check
      server      worker2 192.168.3.52:443 check
      

    We use this approach in OCP LibVirt CI see haproxy_C155F2U31.cfg

    1. named support multiple conf files using the include directive

      1. Create the modular directory: mkdir -p /etc/named/conf.d
      2. Modify /etc/named.conf to include your custom zone files. include "/etc/named/conf.d/ocp-cluster-1.conf";
      3. Create the file /etc/named/conf.d/ocp-cluster-1.conf
      zone "mycluster2.ibm.net" IN {
          type master;
          file "/var/named/zones/db.ocp-cluster-1.local";
          allow-query { any; };
      };
      
      zone "122.168.192.in-addr.arpa" IN {
          type master;
          file "/var/named/zones/ocp-cluster-1.192.168.122";
          allow-query { any; };
      };
      

    Using this approach you’ll be able to share the bastion.

  • OpenShift Container Platform 4.22.0 has been released

    New features are:

    • Installer-provisioned infrastructure for IBM PowerVC is now generally available.
    • Enforce RSA key format for Installer-provisioned infrastructure on IBM Power® Virtual Server.
    • Harden the destroy logic for Installer-provisioned infrastructure on IBM Power® Virtual Server to simplify removing a cluster.
    • RHEL-10 Tech Preview with osImageStream

    Release Notes https://docs.redhat.com/en/documentation/openshift_container_platform/4.22/html/release_notes/ocp-4-22-release-notes#ocp-release-notes-ibm-power_release-notes 

    Video YouTube: What’s New in OpenShift 4.22 – Key Updates and New Features

  • R.reduce is not a function

    My teammate hit this issue `R.reduce is not a function` when using OperatorHub

    Workaround:

    oc patch operatorhubs/cluster --type merge --patch '{"spec":{"sources":[{"disabled": true,"name": "community-operators"},{"disabled": true,"name": "certified-operators}]}}'

    PR #16588 • Bug OCPBUGS-88027

    Thanks to https://access.redhat.com/solutions/7075189 for the solution

  • June 2026: Additions IBM Power Open Source Images on the IBM Container Registry

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). New images for redis-operator and opa (open policy agent) are particular interesting for those working with analytics and caching.

    redis-operator	v0.24.0 	Apache-2.0 	podman pull icr.io/ppc64le-oss/redis-operator-ppc64le:v0.24.0 	May 22, 2026
    opa-ppc64le 	v1.15.1 	Apache-2.0 	podman pull icr.io/ppc64le-oss/opa-ppc64le :v1.15.1 	May 29, 2026
    

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/

  • Power up with omc: The OpenShift Must-Gather Client

    If you are like me, developing code for OpenShift and reviewing logs daily – omc is for you.

    omc is a tool engineers use to inspect resources from an OpenShift must-gather in the same way as they are retrieved with the oc command. It’s a game changer. You can see sophisticated examples at https://github.com/gmeghnag/omc?tab=readme-ov-file#examples

    You can download the tool from github.com:

    1. Windows https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Windows_x86_64.zip
    2. OSX ARM https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Darwin_arm64
    3. OSX Intel https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Darwin_x86_64
    4. Linux ppc64le https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Linux_ppc64le.tar.gz

    The reason this tool matters is you can use this as part of your development tools and I’ve started to see this as part of the tooling.

    Credit to gmeghnag

    Reference: https://access.redhat.com/solutions/6993921

  • REPOST: Using Red Hat Service Interconnect with OpenShift and RHEL on IBM Power

    Original is at https://community.ibm.com/community/user/blogs/paul-bastide/2026/04/27/using-red-hat-service-interconnect-with-ibm-power#ItemCommentPanel

    • Author: Kaushik Talathi, IBM Power
    • Author: Michael Turek, IBM Power
    • Author: Paul Bastide, IBM Power

    As organizations adopt open hybrid cloud and cloud-native architectures, developers face complexity in connecting applications across multiple clouds — public, private, and on-premises systems. Traditional VPNs and firewall rules require extensive network planning, taking days or weeks to deploy, which delay development and project delivery.

    Red Hat Service Interconnect enables developers to create secure Layer-7 connections on-demand. Based on the open source Skupper project, Red Hat Service Interconnect enables application connectivity across Red Hat Enterprise Linux, Red Hat OpenShift Container Platform clusters, and non-Red Hat environments. Your application are able to talk to each other as if they were on the same local network — without complex VPNs or firewall headaches.

    With the release of Red Hat Service Interconnect (RHSI) v2.1.2, RHSI runs on IBM Power Systems and using a simple CLI, your workloads can seamlessly join your cross-architecture service mesh in minutes – no extensive networking planning or added security risk.

    This document walks you through the installation and setup, and a hello world to ease your adoption of RHSI.

    Installation and Setup

    Installing RHSI Operator using the OpenShift Console

    To install the RHSI on your OpenShift Container Platform 4.18 or higher system, go to the OperatorHub:

    1. Login with a user id that has cluster-admin user access
    2. In the OpenShift Container Platform web console, navigate to EcosystemSoftware Catalog.
    3. Search for the Red Hat Service Interconnect, then click Install.
    4. Keep the default selection of Installation mode and namespace to ensure that the Operator will be installed to the openshift-operators namespace.
    5. Click Install.
    6. Repeat the same process for Red Hat Service Interconnect Network Observer.
    7. Verify the installation succeeded by inspecting the ClusterServiceVersion (csv) file:
    $ oc project openshift-operators
    
    $ oc get csv
    NAME                                  DISPLAY                                         VERSION      REPLACES                              PHASE
    skupper-netobs-operator.v2.1.3-rh-1   Red Hat Service Interconnect Network Observer   2.1.3-rh-1   skupper-netobs-operator.v2.1.2-rh-2   Succeeded
    skupper-operator.v2.1.3-rh-1          Red Hat Service Interconnect                    2.1.3-rh-1   skupper-operator.v2.1.2-rh-2          Succeeded
    
    1. Verify that the Red Hat Service Interconnect (RHSI) is up and running
    $ oc get deploy -n openshift-operators
    NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
    skupper-controller                           1/1     1            1           9m59s
    skupper-netobs-operator-controller-manager   1/1     1            1           101s
    
    1. Check the pods created for Red Hat Service Interconnect (RHSI) through the command line interface:
    $ oc get pods
    NAME                                                         READY   STATUS    RESTARTS   AGE
    skupper-controller-779d985989-vqvvb                          1/1     Running   0          11m
    skupper-netobs-operator-controller-manager-85957676f-p98tc   1/1     Running   0          3m40s
    

    Installing Red Hat Service Interconnect CLI

    To install RHSI CLI on OCP Bastion Node and Linux System, enable Red Hat package:

    1. Use the subscription-manager command to subscribe to the required package repositories.

    Red Hat Enterprise Linux 8

    $ sudo subscription-manager repos --enable=service-interconnect-2-for-rhel-8-ppc64le-rpms
    

    Red Hat Enterprise Linux 9

    $ sudo subscription-manager repos --enable=service-interconnect-2-for-rhel-9-ppc64le-rpms
    
    1. Use yum or dnf commands to install the RHSI CLI & Router.
    $ sudo dnf install skupper-cli skupper-router
    
    1. Verify that CLI & Router is installed correctly.
    $ skupper version
    Warning: Docker is not installed. Skipping image digests search.
    COMPONENT               VERSION
    router                  3.4.2
    network-observer        2.1.3
    cli                     2.1.3
    system-controller       2.1.3
    prometheus              v4.16.0
    origin-oauth-proxy      v4.16.0
    

    Hello World Example

    To show RHSI in action, you need an application to use – HTTP Hello World application with a frontend and backend service. The frontend uses the backend to process requests. In this scenario, the backend is deployed in the hello-world-east namespace of rhsi-east cluster and the frontend is deployed in the hello-world-west namespace of rhsi-west another cluster as well as local-west namespace on a RHEL system. You are able to use multiple namespaces, typically on different clusters or from a single machine.

    1. Configure access to multiple namespaces on OCP Clusters and Local Systems

    1. Start a console session for each of your namespaces. Set the KUBECONFIG environment variable to a different path in each session. For the Local System, ensure skupper CLI is installed.
    ## Console for West cluster
    $ export KUBECONFIG=$HOME/.kube/config-hello-world-west
    
    ## Console for East cluster
    $ export KUBECONFIG=$HOME/.kube/config-hello-world-east
    
    ## Local System
    $ systemctl --user enable --now podman.socket
    $ loginctl enable-linger <username>
    $ export REGISTRY_AUTH_FILE=/path/to/auth-file
    $ export SKUPPER_PLATFORM=podman
    $ podman login registry.Red Hat.io
    $ skupper system install
    Platform podman is now configured for Skupper
    
    1. Create and set the namespaces.
    ## Console for West cluster
    $ oc create namespace hello-world-west
    $ oc config set-context --current --namespace hello-world-west
    
    ## Console for East cluster
    $ oc create namespace hello-world-east
    $ oc config set-context --current --namespace hello-world-east
    

    2. Creating Sites on Clusters and Local System

    1. Create Site with link access enabled for external connections using earlier set context.
    ## West Cluster
    apiVersion: skupper.io/v2alpha1
    kind: Site
    metadata:
      name: west
      namespace: hello-world-west
    spec:
      linkAccess: default
    
    ## East Cluster
    apiVersion: skupper.io/v2alpha1
    kind: Site
    metadata:
      name: east
      namespace: hello-world-east
    spec:
      linkAccess: default
    
    ## Local System
    $ skupper site create local-west-site -n local-west --enable-link-access
    File written to /var/lib/skupper/namespaces/local-west/input/resources/Site-local-west-site.yaml
    File written to /var/lib/skupper/namespaces/local-west/input/resources/RouterAccess-router-access-local-west-site.yaml
    
    $ skupper system start -n local-west
    Sources will be consumed from namespace "local-west"
    Site "local-west-site" has been created on namespace "local-west"
    Platform: podman
    Static links have been defined at: /var/lib/skupper/namespaces/local-west/runtime/links
    Definition is available at: /var/lib/skupper/namespaces/local-west/input/resources
    
    1. Validate sites are in Ready state.
    ## West Cluster
    $ oc get site
    NAME   STATUS    SITES IN NETWORK   MESSAGE
    west   Pending                      containers with unready status: [router kube-adaptor]
    
    $ oc get site
    NAME   STATUS   SITES IN NETWORK   MESSAGE
    west   Ready    1                  OK
    
    
    ## East Cluster
    $ oc get site
    NAME   STATUS    SITES IN NETWORK   MESSAGE
    west   Pending                      containers with unready status: [router kube-adaptor]
    
    $ oc get site
    NAME   STATUS   SITES IN NETWORK   MESSAGE
    west   Ready    1                  OK
    
    ## Local System
    $ skupper site status -n local-west
    NAME            STATUS  MESSAGE
    local-west-site Ready   OK
    

    The message containers with unready status: [router kube-adaptor] is expected.

    3. Linking Sites

    Once sites are linked, services can be exposed and consumed across the application network without the need to open ports or manage inter-site connectivity. You’ll find that there are two key types of link connection:

    • Connecting site: The site that initiates the link connection.
    • Listening site: The site receives the link connection.

    The link direction is not significant, and is typically determined by ease of connectivity. For example, if east site is behind a firewall and west site is a cluster on the public cloud, linking from east to west sites is the easiest option.

    AccessGrant – Permission on a listening site enabling access token redemption to create links. Grants access to the GrantServer (HTTPS server) which provides a URL, secret code, and cert bundled into an AccessToken. Token redemption limits and duration are configurable. Exposed via Route (OpenShift) or LoadBalancer (other systems).

    AccessToken – Short-lived, typically single-use credential containing the GrantServer URL, secret code, and cert. A connecting site redeems this token to establish a link to the listening site.

    To link sites, AccessGrant and AccessToken resources on the listening site and apply the AccessToken resource on the connecting site to create the link.

    1. On the listening(for example west) site, create an AccessGrant resource for Kubenetes connecting east site. For local-system linking, generate a link resource on kubernetes cluster site – for example east – where system site needs to be connected.
    ## West Cluster to East cluster
    apiVersion: skupper.io/v2alpha1
    kind: AccessGrant
    metadata:
      name: grant-west
    spec:
      redemptionsAllowed: 2        # default 1
      expirationWindow: 25m        # default 15m
    
    ## East Cluster link to local system
    $ skupper link generate > link-for-local-site.yaml
    
    1. Validate the AccessGrant resource on listening(for example west) site:
    $ oc get accessgrant
    NAME         REDEMPTIONS ALLOWED   REDEMPTIONS MADE   EXPIRATION             STATUS   MESSAGE
    grant-west   2                     0                  2026-04-22T17:11:12Z   Ready    OK
    
    1. On the listening(for example west) site, populate environment variables to allow token generation:
    URL="$(oc get accessgrant grant-west -o template --template '{{ .status.url }}')"
    CODE="$(oc get accessgrant grant-west -o template --template '{{ .status.code }}')"
    CA_RAW="$(oc get accessgrant grant-west -o template --template '{{ .status.ca }}')"
    

    URL is the URL of the GrantServer CODE is the secret code to access the GrantServer CA_RAW is the cert required to establish a HTTPS connection to the GrantServer

    1. On the listening(for example west) site, create a token YAML file named token.yaml.

    NOTE Access to this file provides access to the application network. Protect it appropriately.

    cat > token.yaml <<EOF
    apiVersion: skupper.io/v2alpha1
    kind: AccessToken
    metadata:
      name: token-to-west
    spec:
      code: "$(printf '%s' "$CODE")"
      ca: |- 
    $(printf '%s\n' "$CA_RAW" | sed 's/^/    /')
      url: "$(printf '%s' "$URL")"
    EOF
    
    1. Securely transfer the token.yaml file to context of the connecting(for example east) site, And apply it. For local system, copy the link-for-local-site.yaml file to the local-west site and apply it.
    ## East Cluster
    $ oc apply -f token.yaml
    
    ## Local-west site
    $ skupper system apply -f link-for-local-site.yaml -n local-west
    File written to /var/lib/skupper/namespaces/local-west/input/resources/Link-link-east-skupper-router.yaml
    Link link-east-skupper-router added
    File written to /var/lib/skupper/namespaces/local-west/input/resources/Secret-link-east.yaml
    Secret link-east added
    Custom resources are applied. If a site is already running, run `skupper system reload` to make effective the changes.
    
    $ skupper system reload -n local-west
    Sources will be consumed from namespace "local-west"
    ...
    2026/04/24 12:43:40 WARN certificate will not be overwritten path=/var/lib/skupper/namespaces/local-west/runtime/issuers/skupper-site-ca/tls.key
    Site "local-west-site" has been created on namespace "local-west"
    Platform: podman
    Static links have been defined at: /var/lib/skupper/namespaces/local-west/runtime/links
    Definition is available at: /var/lib/skupper/namespaces/local-west/input/resources
    
    1. On the connecting(for example east & local-west) site, check token and link status: The GrantServer has validated the AccessToken and redeemed it for a Link resource. The connecting site uses Link resource to establish an mTLS connection between routers.
    ## East site
    $ oc get accesstoken
    NAME            URL                                                                                 REDEEMED   STATUS   MESSAGE
    token-to-west   https://<skupper-grant-server-west-site>:443/cc4e6668-1869-4fd9-a9e7-a0a86abbe15d   true       Ready    OK
    
    # oc get link
    NAME            STATUS   REMOTE SITE   MESSAGE
    token-to-west   Ready    west          OK
    
    ## Local-west site 
    $  skupper link status -n local-west
    NAME                            STATUS  COST    MESSAGE
    link-east-skupper-router        Ready   1       OK
    

    4. Exposing services on the application network

    After creating an application network by linking sites, services can be exposed from one site using connectors and consume those services on other sites using listeners.

    1. Create a workload to expose on the network, for example, backend server of hello world example.
    ## East site
    $ oc create deployment backend --image quay.io/skupper/hello-world-backend --replicas 3
    
    ## West site
    $ oc create deployment frontend --image quay.io/skupper/hello-world-frontend
    
    ## Local site
    $ podman run --name frontend --detach --rm -p 9090:8080 quay.io/skupper/hello-w
    orld-frontend
    Trying to pull quay.io/skupper/hello-world-frontend:latest...
    Getting image source signatures
    Copying blob b4c4646a26d4 done   | 
    Copying blob c8939585957e done   | 
    Copying blob b530b5dc825c done   | 
    Copying blob 76789c06b573 done   | 
    Copying blob 10643c2bc08d done   | 
    Copying blob 42c663ca3696 done   | 
    Copying blob 938062c0e7a6 done   | 
    Copying blob 4f2321e928b3 done   | 
    Copying config 75a7a6cc39 done   | 
    Writing manifest to image destination
    84d9a4bd4399ec332faf0f7555278ecdf240ddbf5d4f4773f1fe2893264e933f
    
    1. Create connector resource on east site.
    apiVersion: skupper.io/v2alpha1
    kind: Connector
    metadata:
      name: backend
      namespace: hello-world-east
    spec:
      routingKey: backend
      selector: app=backend
      port: 8080
    
    1. Validate the connector status:
    $ oc get connector
    NAME      ROUTING KEY   PORT   HOST   SELECTOR      STATUS    HAS MATCHING LISTENER   MESSAGE
    backend   backend       8080          app=backend   Pending                           No matching listeners 
    
    1. Create a listener resource on west & local-west site:

    Note Identify a connector that you want to use. Note the routing key of that connector.

    ## West site
    apiVersion: skupper.io/v2alpha1
    kind: Listener
    metadata:
      name: frontend
      namespace: hello-world-west
    spec:
      routingKey: backend
      host: backend
      port: 8080
    
    ## local-west site
    $ skupper listener create local-frontend --routing-key backend  8080 -n local-west
    File written to /var/lib/skupper/namespaces/local-west/input/resources/Listener-local-frontend.yaml
    
    $ skupper system reload -n local-west
    Sources will be consumed from namespace "local-west"
    2026/04/24 12:47:18 WARN certificate will not be overwritten path=/var/lib/skupper/namespaces/local-west/runtime/issuers/skupper-site-ca/tls.crt
    ...
    Site "local-west-site" has been created on namespace "local-west"
    Platform: podman
    Static links have been defined at: /var/lib/skupper/namespaces/local-west/runtime/links
    Definition is available at: /var/lib/skupper/namespaces/local-west/input/resources
    
    1. Validate the listener status:
    ## West site
    $ oc get listener
    NAME       ROUTING KEY   PORT   HOST      STATUS   HAS MATCHING CONNECTOR   MESSAGE
    frontend   backend       8080   backend   Ready    true                     OK
    
    ## local-west site
    $ skupper listener status -n local-west
    NAME            STATUS  ROUTING-KEY     HOST    PORT    MATCHING-CONNECTOR      MESSAGE
    local-frontend  Ready   backend         0.0.0.0 8080    true                    OK
    
    1. Test the Hello World Application

    To test our Hello World, we need external access to the frontend. Use oc port-forward to make the frontend available at localhost:8080.

    ## West site
    $ oc port-forward deployment/frontend 8080:8080 &
    

    If everything is in order, you can now access the web interface by navigating to this URL in your browser http://localhost:8080/

    The frontend assigns each new user a name. Click Say hello to send a greeting to the backend and get a greeting in response.

    For local system tests, you can run following command:

    $ curl -s http://localhost:8080/api/hello  -d '{"name":"Jack Sparrow"}' | jq -r '.text'
    Hi, Jack Sparrow.  I am Astonishing Application (backend-66dbcb9494-t7wlg).
    

    5. Setting up Network Observer

    The console provides a visual overview of the sites, links, services, and communication metrics.

    1. Create NetworkObserver object:
    apiVersion: observability.skupper.io/v2alpha1
    kind: NetworkObserver
    metadata:
      name: networkobserver-sample
      namespace: hello-world-west
    spec: {}
    
    1. Determine the console URL and use this URL to login to skupper console via browser and when prompted login using OCP credentials.
    $ oc get --namespace hello-world-west -o jsonpath="{.spec.host}" route net
    workobserver-sample-network-observer
    <NetworkObserver-URL>
    

    The Skupper console is used to monitor and troubleshoot application network. The console provides a visual overview of the sites, links, services, and communication metrics.

    Sites view

    The Sites tab displays the network topology showing three interconnected sites in the Hello World example: the east site (OCP cluster hosting the backend service), the west site (OCP cluster with frontend service), and local-west-site (RHEL local system running Skupper on Podman with frontend service). The dashed lines represent active links connecting these sites, enabling frontend service from different environments to access the backend service seamlessly.

    Components view

    The Components tab shows the logical service architecture with three key elements: hello-world-frontend & local system site, consuming service and hello-world-backend, an exposed service. The directional arrow illustrates how the frontend component communicates with the backend through the Skupper service network demonstrating cross-site service connectivity.

    Processes view

    The Processes tab displays actual running pods and real-time traffic metrics, showing backend service in the east site on OCP cluster processing requests from two frontend clients in the west site on OCP cluster with total of 2.1 KB traffic and local-weat-site on the local linux system with podman showcasing 2.9 KB traffic volume. This validates that RHSI setups can be configured across the hybrid cloud setup.

    Cleaning up

    To remove Skupper and the other resources from this exercise, use the following commands:

    ## West site
    $ skupper site delete --all
    $ oc delete deployment/frontend
    
    ## East
    $ skupper site delete --all
    $ oc delete deployment/backend
    
    ## Local System
    $ skupper system stop -n local-west
    

    Conclusion

    Red Hat Service Interconnect (RHSI) simplifies hybrid cloud connectivity by enabling secure, on-demand application connections across diverse environments without complex VPNs or firewall headaches. RHSI support on IBM Power showcases seamless interconnect between services across OpenShift clusters and local RHEL systems, with the Network Observer console providing real-time visibility into the distributed service mesh.

    Best wishes and good luck with your RHSI journey! 🚀

    References & Additional Resources