Blog

If you are like me, developing code for OpenShift and reviewing logs daily – omc is for you.

omc is a tool engineers use to inspect resources from an OpenShift must-gather in the same way as they are retrieved with the oc command. It’s a game changer. You can see sophisticated examples at https://github.com/gmeghnag/omc?tab=readme-ov-file#examples

You can download the tool from github.com:

1. Windows https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Windows_x86_64.zip
2. OSX ARM https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Darwin_arm64
3. OSX Intel https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Darwin_x86_64
4. Linux ppc64le https://github.com/gmeghnag/omc/releases/download/v3.13.3/omc_Linux_ppc64le.tar.gz

The reason this tool matters is you can use this as part of your development tools and I’ve started to see this as part of the tooling.

Credit to gmeghnag

Reference: https://access.redhat.com/solutions/6993921

Original is at https://community.ibm.com/community/user/blogs/paul-bastide/2026/04/27/using-red-hat-service-interconnect-with-ibm-power#ItemCommentPanel

• Author: Kaushik Talathi, IBM Power
• Author: Michael Turek, IBM Power
• Author: Paul Bastide, IBM Power

As organizations adopt open hybrid cloud and cloud-native architectures, developers face complexity in connecting applications across multiple clouds — public, private, and on-premises systems. Traditional VPNs and firewall rules require extensive network planning, taking days or weeks to deploy, which delay development and project delivery.

Red Hat Service Interconnect enables developers to create secure Layer-7 connections on-demand. Based on the open source Skupper project, Red Hat Service Interconnect enables application connectivity across Red Hat Enterprise Linux, Red Hat OpenShift Container Platform clusters, and non-Red Hat environments. Your application are able to talk to each other as if they were on the same local network — without complex VPNs or firewall headaches.

With the release of Red Hat Service Interconnect (RHSI) v2.1.2, RHSI runs on IBM Power Systems and using a simple CLI, your workloads can seamlessly join your cross-architecture service mesh in minutes – no extensive networking planning or added security risk.

This document walks you through the installation and setup, and a hello world to ease your adoption of RHSI.

Installation and Setup

Installing RHSI Operator using the OpenShift Console

To install the RHSI on your OpenShift Container Platform 4.18 or higher system, go to the OperatorHub:

1. Login with a user id that has cluster-admin user access
2. In the OpenShift Container Platform web console, navigate to Ecosystem → Software Catalog.
3. Search for the Red Hat Service Interconnect, then click Install.
4. Keep the default selection of Installation mode and namespace to ensure that the Operator will be installed to the openshift-operators namespace.
5. Click Install.
6. Repeat the same process for Red Hat Service Interconnect Network Observer.
7. Verify the installation succeeded by inspecting the ClusterServiceVersion (csv) file:

$ oc project openshift-operators

$ oc get csv
NAME                                  DISPLAY                                         VERSION      REPLACES                              PHASE
skupper-netobs-operator.v2.1.3-rh-1   Red Hat Service Interconnect Network Observer   2.1.3-rh-1   skupper-netobs-operator.v2.1.2-rh-2   Succeeded
skupper-operator.v2.1.3-rh-1          Red Hat Service Interconnect                    2.1.3-rh-1   skupper-operator.v2.1.2-rh-2          Succeeded

8. Verify that the Red Hat Service Interconnect (RHSI) is up and running

$ oc get deploy -n openshift-operators
NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
skupper-controller                           1/1     1            1           9m59s
skupper-netobs-operator-controller-manager   1/1     1            1           101s

8. Check the pods created for Red Hat Service Interconnect (RHSI) through the command line interface:

$ oc get pods
NAME                                                         READY   STATUS    RESTARTS   AGE
skupper-controller-779d985989-vqvvb                          1/1     Running   0          11m
skupper-netobs-operator-controller-manager-85957676f-p98tc   1/1     Running   0          3m40s

Installing Red Hat Service Interconnect CLI

To install RHSI CLI on OCP Bastion Node and Linux System, enable Red Hat package:

1. Use the subscription-manager command to subscribe to the required package repositories.

Red Hat Enterprise Linux 8

$ sudo subscription-manager repos --enable=service-interconnect-2-for-rhel-8-ppc64le-rpms

Red Hat Enterprise Linux 9

$ sudo subscription-manager repos --enable=service-interconnect-2-for-rhel-9-ppc64le-rpms

2. Use yum or dnf commands to install the RHSI CLI & Router.

$ sudo dnf install skupper-cli skupper-router

3. Verify that CLI & Router is installed correctly.

$ skupper version
Warning: Docker is not installed. Skipping image digests search.
COMPONENT               VERSION
router                  3.4.2
network-observer        2.1.3
cli                     2.1.3
system-controller       2.1.3
prometheus              v4.16.0
origin-oauth-proxy      v4.16.0

Hello World Example

To show RHSI in action, you need an application to use – HTTP Hello World application with a frontend and backend service. The frontend uses the backend to process requests. In this scenario, the backend is deployed in the hello-world-east namespace of rhsi-east cluster and the frontend is deployed in the hello-world-west namespace of rhsi-west another cluster as well as local-west namespace on a RHEL system. You are able to use multiple namespaces, typically on different clusters or from a single machine.

1. Configure access to multiple namespaces on OCP Clusters and Local Systems

1. Start a console session for each of your namespaces. Set the KUBECONFIG environment variable to a different path in each session. For the Local System, ensure skupper CLI is installed.

## Console for West cluster
$ export KUBECONFIG=$HOME/.kube/config-hello-world-west

## Console for East cluster
$ export KUBECONFIG=$HOME/.kube/config-hello-world-east

## Local System
$ systemctl --user enable --now podman.socket
$ loginctl enable-linger <username>
$ export REGISTRY_AUTH_FILE=/path/to/auth-file
$ export SKUPPER_PLATFORM=podman
$ podman login registry.Red Hat.io
$ skupper system install
Platform podman is now configured for Skupper

3. Create and set the namespaces.

## Console for West cluster
$ oc create namespace hello-world-west
$ oc config set-context --current --namespace hello-world-west

## Console for East cluster
$ oc create namespace hello-world-east
$ oc config set-context --current --namespace hello-world-east

2. Creating Sites on Clusters and Local System

1. Create Site with link access enabled for external connections using earlier set context.

## West Cluster
apiVersion: skupper.io/v2alpha1
kind: Site
metadata:
  name: west
  namespace: hello-world-west
spec:
  linkAccess: default

## East Cluster
apiVersion: skupper.io/v2alpha1
kind: Site
metadata:
  name: east
  namespace: hello-world-east
spec:
  linkAccess: default

## Local System
$ skupper site create local-west-site -n local-west --enable-link-access
File written to /var/lib/skupper/namespaces/local-west/input/resources/Site-local-west-site.yaml
File written to /var/lib/skupper/namespaces/local-west/input/resources/RouterAccess-router-access-local-west-site.yaml

$ skupper system start -n local-west
Sources will be consumed from namespace "local-west"
Site "local-west-site" has been created on namespace "local-west"
Platform: podman
Static links have been defined at: /var/lib/skupper/namespaces/local-west/runtime/links
Definition is available at: /var/lib/skupper/namespaces/local-west/input/resources

2. Validate sites are in Ready state.

## West Cluster
$ oc get site
NAME   STATUS    SITES IN NETWORK   MESSAGE
west   Pending                      containers with unready status: [router kube-adaptor]

$ oc get site
NAME   STATUS   SITES IN NETWORK   MESSAGE
west   Ready    1                  OK


## East Cluster
$ oc get site
NAME   STATUS    SITES IN NETWORK   MESSAGE
west   Pending                      containers with unready status: [router kube-adaptor]

$ oc get site
NAME   STATUS   SITES IN NETWORK   MESSAGE
west   Ready    1                  OK

## Local System
$ skupper site status -n local-west
NAME            STATUS  MESSAGE
local-west-site Ready   OK

The message containers with unready status: [router kube-adaptor] is expected.

3. Linking Sites

Once sites are linked, services can be exposed and consumed across the application network without the need to open ports or manage inter-site connectivity. You’ll find that there are two key types of link connection:

• Connecting site: The site that initiates the link connection.
• Listening site: The site receives the link connection.

The link direction is not significant, and is typically determined by ease of connectivity. For example, if east site is behind a firewall and west site is a cluster on the public cloud, linking from east to west sites is the easiest option.

AccessGrant – Permission on a listening site enabling access token redemption to create links. Grants access to the GrantServer (HTTPS server) which provides a URL, secret code, and cert bundled into an AccessToken. Token redemption limits and duration are configurable. Exposed via Route (OpenShift) or LoadBalancer (other systems).

AccessToken – Short-lived, typically single-use credential containing the GrantServer URL, secret code, and cert. A connecting site redeems this token to establish a link to the listening site.

To link sites, AccessGrant and AccessToken resources on the listening site and apply the AccessToken resource on the connecting site to create the link.

1. On the listening(for example west) site, create an AccessGrant resource for Kubenetes connecting east site. For local-system linking, generate a link resource on kubernetes cluster site – for example east – where system site needs to be connected.

## West Cluster to East cluster
apiVersion: skupper.io/v2alpha1
kind: AccessGrant
metadata:
  name: grant-west
spec:
  redemptionsAllowed: 2        # default 1
  expirationWindow: 25m        # default 15m

## East Cluster link to local system
$ skupper link generate > link-for-local-site.yaml

2. Validate the AccessGrant resource on listening(for example west) site:

$ oc get accessgrant
NAME         REDEMPTIONS ALLOWED   REDEMPTIONS MADE   EXPIRATION             STATUS   MESSAGE
grant-west   2                     0                  2026-04-22T17:11:12Z   Ready    OK

3. On the listening(for example west) site, populate environment variables to allow token generation:

URL="$(oc get accessgrant grant-west -o template --template '{{ .status.url }}')"
CODE="$(oc get accessgrant grant-west -o template --template '{{ .status.code }}')"
CA_RAW="$(oc get accessgrant grant-west -o template --template '{{ .status.ca }}')"

URL is the URL of the GrantServer CODE is the secret code to access the GrantServer CA_RAW is the cert required to establish a HTTPS connection to the GrantServer

4. On the listening(for example west) site, create a token YAML file named token.yaml.

NOTE Access to this file provides access to the application network. Protect it appropriately.

cat > token.yaml <<EOF
apiVersion: skupper.io/v2alpha1
kind: AccessToken
metadata:
  name: token-to-west
spec:
  code: "$(printf '%s' "$CODE")"
  ca: |- 
$(printf '%s\n' "$CA_RAW" | sed 's/^/    /')
  url: "$(printf '%s' "$URL")"
EOF

5. Securely transfer the token.yaml file to context of the connecting(for example east) site, And apply it. For local system, copy the link-for-local-site.yaml file to the local-west site and apply it.

## East Cluster
$ oc apply -f token.yaml

## Local-west site
$ skupper system apply -f link-for-local-site.yaml -n local-west
File written to /var/lib/skupper/namespaces/local-west/input/resources/Link-link-east-skupper-router.yaml
Link link-east-skupper-router added
File written to /var/lib/skupper/namespaces/local-west/input/resources/Secret-link-east.yaml
Secret link-east added
Custom resources are applied. If a site is already running, run `skupper system reload` to make effective the changes.

$ skupper system reload -n local-west
Sources will be consumed from namespace "local-west"
...
2026/04/24 12:43:40 WARN certificate will not be overwritten path=/var/lib/skupper/namespaces/local-west/runtime/issuers/skupper-site-ca/tls.key
Site "local-west-site" has been created on namespace "local-west"
Platform: podman
Static links have been defined at: /var/lib/skupper/namespaces/local-west/runtime/links
Definition is available at: /var/lib/skupper/namespaces/local-west/input/resources

6. On the connecting(for example east & local-west) site, check token and link status: The GrantServer has validated the AccessToken and redeemed it for a Link resource. The connecting site uses Link resource to establish an mTLS connection between routers.

## East site
$ oc get accesstoken
NAME            URL                                                                                 REDEEMED   STATUS   MESSAGE
token-to-west   https://<skupper-grant-server-west-site>:443/cc4e6668-1869-4fd9-a9e7-a0a86abbe15d   true       Ready    OK

# oc get link
NAME            STATUS   REMOTE SITE   MESSAGE
token-to-west   Ready    west          OK

## Local-west site 
$  skupper link status -n local-west
NAME                            STATUS  COST    MESSAGE
link-east-skupper-router        Ready   1       OK

4. Exposing services on the application network

After creating an application network by linking sites, services can be exposed from one site using connectors and consume those services on other sites using listeners.

1. Create a workload to expose on the network, for example, backend server of hello world example.

## East site
$ oc create deployment backend --image quay.io/skupper/hello-world-backend --replicas 3

## West site
$ oc create deployment frontend --image quay.io/skupper/hello-world-frontend

## Local site
$ podman run --name frontend --detach --rm -p 9090:8080 quay.io/skupper/hello-w
orld-frontend
Trying to pull quay.io/skupper/hello-world-frontend:latest...
Getting image source signatures
Copying blob b4c4646a26d4 done   | 
Copying blob c8939585957e done   | 
Copying blob b530b5dc825c done   | 
Copying blob 76789c06b573 done   | 
Copying blob 10643c2bc08d done   | 
Copying blob 42c663ca3696 done   | 
Copying blob 938062c0e7a6 done   | 
Copying blob 4f2321e928b3 done   | 
Copying config 75a7a6cc39 done   | 
Writing manifest to image destination
84d9a4bd4399ec332faf0f7555278ecdf240ddbf5d4f4773f1fe2893264e933f

2. Create connector resource on east site.

apiVersion: skupper.io/v2alpha1
kind: Connector
metadata:
  name: backend
  namespace: hello-world-east
spec:
  routingKey: backend
  selector: app=backend
  port: 8080

3. Validate the connector status:

$ oc get connector
NAME      ROUTING KEY   PORT   HOST   SELECTOR      STATUS    HAS MATCHING LISTENER   MESSAGE
backend   backend       8080          app=backend   Pending                           No matching listeners 

4. Create a listener resource on west & local-west site:

Note Identify a connector that you want to use. Note the routing key of that connector.

## West site
apiVersion: skupper.io/v2alpha1
kind: Listener
metadata:
  name: frontend
  namespace: hello-world-west
spec:
  routingKey: backend
  host: backend
  port: 8080

## local-west site
$ skupper listener create local-frontend --routing-key backend  8080 -n local-west
File written to /var/lib/skupper/namespaces/local-west/input/resources/Listener-local-frontend.yaml

$ skupper system reload -n local-west
Sources will be consumed from namespace "local-west"
2026/04/24 12:47:18 WARN certificate will not be overwritten path=/var/lib/skupper/namespaces/local-west/runtime/issuers/skupper-site-ca/tls.crt
...
Site "local-west-site" has been created on namespace "local-west"
Platform: podman
Static links have been defined at: /var/lib/skupper/namespaces/local-west/runtime/links
Definition is available at: /var/lib/skupper/namespaces/local-west/input/resources

5. Validate the listener status:

## West site
$ oc get listener
NAME       ROUTING KEY   PORT   HOST      STATUS   HAS MATCHING CONNECTOR   MESSAGE
frontend   backend       8080   backend   Ready    true                     OK

## local-west site
$ skupper listener status -n local-west
NAME            STATUS  ROUTING-KEY     HOST    PORT    MATCHING-CONNECTOR      MESSAGE
local-frontend  Ready   backend         0.0.0.0 8080    true                    OK

6. Test the Hello World Application

To test our Hello World, we need external access to the frontend. Use oc port-forward to make the frontend available at localhost:8080.

## West site
$ oc port-forward deployment/frontend 8080:8080 &

If everything is in order, you can now access the web interface by navigating to this URL in your browser http://localhost:8080/

The frontend assigns each new user a name. Click Say hello to send a greeting to the backend and get a greeting in response.

For local system tests, you can run following command:

$ curl -s http://localhost:8080/api/hello  -d '{"name":"Jack Sparrow"}' | jq -r '.text'
Hi, Jack Sparrow.  I am Astonishing Application (backend-66dbcb9494-t7wlg).

5. Setting up Network Observer

The console provides a visual overview of the sites, links, services, and communication metrics.

1. Create NetworkObserver object:

apiVersion: observability.skupper.io/v2alpha1
kind: NetworkObserver
metadata:
  name: networkobserver-sample
  namespace: hello-world-west
spec: {}

2. Determine the console URL and use this URL to login to skupper console via browser and when prompted login using OCP credentials.

$ oc get --namespace hello-world-west -o jsonpath="{.spec.host}" route net
workobserver-sample-network-observer
<NetworkObserver-URL>

The Skupper console is used to monitor and troubleshoot application network. The console provides a visual overview of the sites, links, services, and communication metrics.

Sites view

The Sites tab displays the network topology showing three interconnected sites in the Hello World example: the east site (OCP cluster hosting the backend service), the west site (OCP cluster with frontend service), and local-west-site (RHEL local system running Skupper on Podman with frontend service). The dashed lines represent active links connecting these sites, enabling frontend service from different environments to access the backend service seamlessly.

Components view

The Components tab shows the logical service architecture with three key elements: hello-world-frontend & local system site, consuming service and hello-world-backend, an exposed service. The directional arrow illustrates how the frontend component communicates with the backend through the Skupper service network demonstrating cross-site service connectivity.

Processes view

The Processes tab displays actual running pods and real-time traffic metrics, showing backend service in the east site on OCP cluster processing requests from two frontend clients in the west site on OCP cluster with total of 2.1 KB traffic and local-weat-site on the local linux system with podman showcasing 2.9 KB traffic volume. This validates that RHSI setups can be configured across the hybrid cloud setup.

Cleaning up

To remove Skupper and the other resources from this exercise, use the following commands:

## West site
$ skupper site delete --all
$ oc delete deployment/frontend

## East
$ skupper site delete --all
$ oc delete deployment/backend

## Local System
$ skupper system stop -n local-west

Conclusion

Red Hat Service Interconnect (RHSI) simplifies hybrid cloud connectivity by enabling secure, on-demand application connections across diverse environments without complex VPNs or firewall headaches. RHSI support on IBM Power showcases seamless interconnect between services across OpenShift clusters and local RHEL systems, with the Network Observer console providing real-time visibility into the distributed service mesh.

Best wishes and good luck with your RHSI journey! 🚀

References & Additional Resources

• Red Hat Service Interconnect usage documentations
• Red Hat Service Interconnect Product Page
• Introducing Red Hat Service Interconnect – Blog Post by Mike Ward
• Skupper v1 Documentations
• Skupper v2 Documentations
• Skupper CLI Documentations
• Skupper GitHub Repository

We are excited to announce with the release of Red Hat Service Interconnect (RHSI) v2.1.2 runs on IBM Power Systems and workloads can now seamlessly join your cross-architecture service mesh!

Red Hat Service Interconnect is based on the Skupper project, allowing you to create a Layer-7 service interconnect across different clouds and clusters. It allows your apps to talk to each other as if they were on the same local network — without complex VPNs or firewall headaches.

A few key things to know about RHSI:

• Security First All traffic is encrypted automatically using mTLS.
• No Root Needed Operates at the application layer; no cluster-admin rights required to get started.
• Seamless Integration Easily connect a frontend in the public cloud to a database on a Power system in your private datacenter.

Check out the official release notes and supported configurations here:
https://docs.redhat.com/en/documentation/red_hat_service_interconnect/2.1/html/release_notes/supported-configurations

Optimizing infrastructure costs starts with understanding how your software licensing interacts with your hardware capabilities. In his updated blog post, IBM’s Maarten Kreuger breaks down the nuances of Red Hat OpenShift subscriptions specifically for IBM Power systems.

The post explores how the unique features of the Power Hypervisor (PowerVM) allow for highly granular licensing. Whether you are using dedicated cores or leveraging Shared Processor LPARs, understanding the math behind “core-pairs”, bare metal and “socket models” is essential for a cost-effective deployment.

Key highlights from the blog include:

• The Power Advantage: How PowerVM’s hardware-enforced hypervisor allows for per-core licensing and fine-grained increments (as small as 0.05 cores).
• Subscription Models: A comparison between the simple Socket Model (ideal for scale-out servers like the S1122) and the Core-Pair Model (best for shared or co-hosted environments).
• The SMT Variable: Why SMT (Simultaneous Multi-Threading) doesn’t increase your license costs, despite reporting more vCPUs.
• Optimization Tactics: How to use Shared Processor Pools to cap CPU usage and prevent paying for the same physical core twice.

Whether you’re running a single cluster or managing complex Power Enterprise Pools 2.0, this guide provides the clarity needed to ensure you aren’t over-subscribing.

Read the full technical deep-dive at OpenShift Subscriptions on Power

The following is a research project I investigated… and notes on what I would do, saving for others to take advantage of:

To demonstrate RoCE (RDMA over Converged Ethernet) usage across nodes on Red Hat OpenShift, you need a container image that includes the RDMA core libraries, OFED drivers, and performance testing tools like perftest (which provides ib_write_bw, ib_send_lat, etc.).

Based on the Red Hat learning path you provided, here is a optimized Podman/Docker Dockerfile and the necessary configuration to run it.

1. The Podman/Docker image

This Dockerfile uses Red Hat Universal Base Image (UBI) 9 and installs the essential RDMA stack and the perftest suite.

# Use RHEL 9 UBI as the base
FROM registry.access.redhat.com/ubi9/ubi:latest

LABEL maintainer="OpenShift RoCE Demo"

# Install RDMA core libraries, drivers, and performance testing tools
# 'perftest' contains the ib_write_bw, ib_read_bw, etc. commands
RUN dnf install -y \
    libibverbs \
    libibverbs-utils \
    rdma-core \
    iproute \
    pciutils \
    ethtool \
    perftest \
    && dnf clean all

# Set working directory
WORKDIR /root

# Default command to keep the container running so you can 'exec' into it
CMD ["sleep", "infinity"]

2. Build and Push the Image

Use Podman to build the image and push it to a registry accessible by your OpenShift cluster (e.g., Quay.io or your internal OpenShift registry).

# Build the image
podman build -t quay.io/<your-username>/roce-test:latest .

# Push the image
podman push quay.io/<your-username>/roce-test:latest

3. Demonstrating Cross-Node Usage (The Test)

To prove RoCE is working across nodes, you must bypass the standard SDN (Software Defined Network) by using Host Networking or a Secondary Network (Multus). For a quick demonstration, we use hostNetwork: true.

Step A: Deploy two Pods on different nodes

Create a file named roce-demo.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: roce-server
  labels:
    app: roce-test
spec:
  hostNetwork: true # Required to access the host's RDMA/RoCE hardware
  containers:
  - name: main
    image: quay.io/<your-username>/roce-test:latest
    securityContext:
      privileged: true # Required for RDMA device access
---
apiVersion: v1
kind: Pod
metadata:
  name: roce-client
  labels:
    app: roce-test
spec:
  hostNetwork: true
  containers:
  - name: main
    image: quay.io/<your-username>/roce-test:latest
    securityContext:
      privileged: true

Step B: Run the Performance Benchmark

1. Identify the IP of the Server Node:

oc get pod roce-server -o wide
# Note the IP (since it's hostNetwork, this is the Node's IP)

2. Start the Server:

oc exec -it roce-server -- ib_write_bw -d <rdma_device_name> -a

(Note: Use ibv_devinfo inside the pod to find your device name, e.g., mlx5_0) 3. Run the Client (from the other pod):

oc exec -it roce-client -- ib_write_bw -d <rdma_device_name> <server_ip> -a

How this demonstrates RoCE:

• Zero-Copy: The ib_write_bw tool performs memory-to-memory transfers without involving the CPU’s TCP/IP stack.
• Performance: If RoCE is correctly configured in your OpenShift cluster (via the Node Network Configuration Policy), you will see bandwidth near the line rate (e.g., ~95Gbps on a 100G link) with extremely low latency compared to standard Ethernet.
• Verification: You can run ethtool -S <interface> on the host while the test is running to see the rdma_ counters increasing, confirming the traffic is not using standard TCP.