Author: Paul

Red Hat OpenShift 4.16 is generally available for upgrades and new installations, and as of today is announced. It is based on Kubernetes 1.29 with the CRI-O 1.29 runtime, RHEL CoreOS 9.4. You can read the release notes at https://docs.openshift.com/container-platform/4.16/release_notes/ocp-4-16-release-notes.html

Some cool features you can use are:

– oc adm upgrade status command, which decouples status information from the existing oc adm upgrade command and provides specific information regarding a cluster update, including the status of the control plane and worker node updates. https://docs.openshift.com/container-platform/4.16/updating/updating_a_cluster/updating-cluster-cli.html#update-upgrading-oc-adm-upgrade-status_updating-cluster-cli

– Tech Preview and Generally Available Table – https://docs.openshift.com/container-platform/4.16/release_notes/ocp-4-16-release-notes.html#ocp-4-16-technology-preview-tables_release-notes

Original post was at https://community.ibm.com/community/user/powerdeveloper/blogs/paul-bastide/2024/04/26/getting-started-with-a-sock-shop-a-sample-multi-ar?CommunityKey=daf9dca2-95e4-4b2c-8722-03cd2275ab63

I’ve developed the following script to help you get started deploying multiarchitecture applications and show elaborate on the techniques for controllin multiarch compute. This script uses the sock-shop application which is available at https://github.com/ocp-power-demos/sock-shop-demo . This series of instructions for sock-shop-demo requires kustomize and following the readme.md in the repository to setup the username and password for mongodb.

You do not need to do every step that follows, please feel free to install/use what you’d like. I recommend the kustomize install with multi-no-ns, and then playing with the features you find interesting. Note, multi-no-ns requires no namespace.

The layout of the application is described in this diagram:

Deploying a non-multiarch Intel App

This deployment shows the Exec errors and pod scheduling errors that are encountered when scheduling Intel only Pods on Power.

For these steps, you are going to clone the ocp-power-demos’s sock-shop-demo and then experiment to resolve errors so the application is up and running.

I’d recommend running this from a bastion.

1. Clone the repository

git clone https://github.com/ocp-power-demos/sock-shop-demo

2. Switch to the sock-shop-demo folder
3. Download kustomize – this tool enable a ordered layout of the resources. You’ll also need oc installed.

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash

Ref: https://kubectl.docs.kubernetes.io/installation/kustomize/binaries/

The reason kustomize is used is due to the sort order feature in the binary.

4. Update the manifests/overlays/single/env.secret file with a username and password for mongodb. openssl rand -hex 10 is a good tip to generating a random password. You’ll need to copy this env.secret in each ‘overlays/` folder that is used in the demo.
5. We’re going to create the sock-shop application.

❯ kustomize build manifests/overlays/single | oc apply -f -

This create a full application within the OpenShift project.

To see the layout of the application you can see the third diagram of the layout (except these are only Intel images) https://github.com/ocp-power-demos/sock-shop-demo/blob/main/README.md#diagrams

6. Run oc get pods -owide

❯ oc get pods -owide
NAME                            READY   STATUS             RESTARTS        AGE     IP            NODE                                   NOMINATED NODE   READINESS GATES
carts-585dc6c878-wq6jg          0/1     Error              6 (2m56s ago)   6m21s   10.129.2.24   mac-01a7-worker-0                      <none>           <none>
carts-db-78f756b87c-r4pl9       1/1     Running            0               6m19s   10.131.0.32   rdr-mac-cust-el-tmwmg-worker-1-6g97b   <none>           <none>
catalogue-77d7c444bb-wnltt      0/1     CrashLoopBackOff   6 (8s ago)      6m17s   10.130.2.21   mac-01a7-worker-1                      <none>           <none>
catalogue-db-5bc97c6b98-v9rdp   1/1     Running            0               6m16s   10.131.0.33   rdr-mac-cust-el-tmwmg-worker-1-6g97b   <none>           <none>
front-end-648fdf6957-bjk9m      0/1     CrashLoopBackOff   5 (2m44s ago)   6m14s   10.129.2.25   mac-01a7-worker-0                      <none>           <none>
orders-5dbf8994df-whb9r         0/1     CrashLoopBackOff   5 (2m47s ago)   6m13s   10.130.2.22   mac-01a7-worker-1                      <none>           <none>
orders-db-7544dc7fd9-w9zh7      1/1     Running            0               6m11s   10.128.3.83   rdr-mac-cust-el-tmwmg-worker-2-5hbxg   <none>           <none>
payment-6cdff467b9-n2dql        0/1     Error              6 (2m53s ago)   6m10s   10.130.2.23   mac-01a7-worker-1                      <none>           <none>
queue-master-c9dcf8f87-c8drl    0/1     CrashLoopBackOff   5 (2m41s ago)   6m8s    10.129.2.26   mac-01a7-worker-0                      <none>           <none>
rabbitmq-54689956b9-rt5fb       2/2     Running            0               6m7s    10.131.0.34   rdr-mac-cust-el-tmwmg-worker-1-6g97b   <none>           <none>
session-db-7d4cc56465-dcx9f     1/1     Running            0               6m5s    10.130.2.24   mac-01a7-worker-1                      <none>           <none>
shipping-5ff5f44465-tbjv7       0/1     Error              6 (2m51s ago)   6m4s    10.130.2.25   mac-01a7-worker-1                      <none>           <none>
user-64dd65b5b7-49cbd           0/1     CrashLoopBackOff   5 (2m25s ago)   6m3s    10.129.2.27   mac-01a7-worker-0                      <none>           <none>
user-db-7f864c9f5f-jchf6        1/1     Running            0               6m1s    10.131.0.35   rdr-mac-cust-el-tmwmg-worker-1-6g97b   <none>           <none>

You might be lucky enough for the scheduler to assign these to Intel only nodes.

At this point if they are all Running with no restarts, yes it’s running.

7. Grab the external URL

❯ oc get routes                                            
NAME        HOST/PORT                                                      PATH   SERVICES    PORT   TERMINATION     WILDCARD
sock-shop   sock-shop-test-user-4.apps.rdr-mac-cust-d.rdr-xyz.net          front-end   8079   edge/Redirect   None

8. Open a Browser, and navigate around. Try registering a user.

It failed for me.

Cordon Power nodes

The purpose is to cordon the Power Nodes and delete the existing pod so you get the Pod running on the architecture you want. This is only recommended on a dev/test system and on the worker nodes.

1. Find the Power workers

oc get nodes -l kubernetes.io/arch=ppc64le | grep worker

2. For each of the Power, cordon the nodes

oc adm cordon node/<worker>

3. List the front-end app pods

❯ oc get pods -l name=front-end
NAME                         READY   STATUS             RESTARTS       AGE
front-end-648fdf6957-bjk9m   0/1     CrashLoopBackOff   13 (26s ago)   42m

4. Delete the front-end pods.

oc delete pod/front-end-648fdf6957-bjk9m

The app should be running correctly at this point.

Use a Node Selector for the Application

Demonstrate how to use node selector to put the workload on the right nodes.

These microservices use Deployments. We can modify the deployment to use NodeSelectors.

1. Edit the manifests/overlays/single/09-front-end-dep.yaml or oc edit deployment/front-end
2. Find the nodeSelector field and add an architecture limitation using a Node label:

nodeSelector:
  node.openshift.io/os_id: rhcos
  kubernetes.io/arch: amd64

3. If you edited, the file run oc apply -f manifests/overlays/single/09-front-end-dep.yaml
4. List the front-end app pods

❯ oc get pods -l name=front-end
NAME                         READY   STATUS              RESTARTS         AGE
front-end-648fdf6957-bjk9m   0/1     CrashLoopBackOff    14 (2m49s ago)   50m
front-end-7bd476764-t974g    0/1     ContainerCreating   0                40s

5. It may not be ‘Ready’, and you may need to delete the front-end pod on the power node.

oc delete pod/front-end-648fdf6957-bjk9m

Note, you can run the following to run with nodeSelectors.

❯ kustomize build manifests/overlays/single-node-selector | oc delete -f - 
❯ kustomize build manifests/overlays/single-node-selector | oc apply -f - 

Are the pods running on the Intel node?

Uncordon the Power nodes

With the nodeSelector now started, you can uncordon the Power nodes. This is only recommended on a dev/test system and on the worker nodes.

1. Find the Power workers

oc get nodes -l kubernetes.io/arch=ppc64le | grep worker

2. For each of the Power, uncordon the nodes

oc adm uncordon node/<worker>

3. List the front-end app pods

❯ oc get pods -l name=front-end
NAME                         READY   STATUS             RESTARTS       AGE
front-end-6944957cd6-qmhhg      1/1     Running            0           19s

The application should be running. If not, please use:

❯ kustomize build manifests/overlays/single-node-selector | oc delete -f - 
❯ kustomize build manifests/overlays/single-node-selector | oc apply -f - 

The workload should be all on the intel side.

Deploying a multiarch Intel/Power App

With many of these applications, there are architecture specific alternatives. You can run without NodeSelectors to get the workload scheduled where there is support.

To switch to Node selectors use across Power/Intel.

1. Switch to oc project sock-shop
2. Delete the Pods and Recreate (this is a manifest-listed set of images)

❯ kustomize build manifests/overlays/multi-no-ns | oc apply -f -

3. List the app pods

❯ oc get pods -owide

Update the app deployment to use a manifest listed image and remove node selector

We’re going to move one of the applications’ dependencies using rabbitmq. The IBM team has created a port of Redis to ppc64le. link

1. Edit git/sock-shop-demo/manifests/overlays/multi-no-ns/19-rabbitmq-dep.yaml
2. Replace image: kbudde/rabbitmq-exporter on line 32. icr.io/ppc64le-oss/rabbitmq-exporter-ppc64le:1.0.0-RC19
3. Remove the nodeSelector label kubernetes.io/arch: amd64 limitation on line 39
4. Build and replace kustomize build manifests/overlays/multi-no-ns | oc apply -f -
5. Check the Pod is starting/running on Power

❯ oc get pod -l name=rabbitmq -owide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
rabbitmq-65c75db8db-9jqbd   2/2     Running   0          96s   10.130.2.31   mac-01a7-worker-1   <none>           <none>

The pod should now start on the Power node.

You’ve taken advantage of the containers, and you can take advantage of other OpenSource container images https://community.ibm.com/community/user/powerdeveloper/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr

Using a Taint/Toleration

Taints and Tolerations provide a way to

1. Find the Secondary workers, for instance, if the primary architecture is Intel, you’ll get the power workers, and taint the Power workers

oc get nodes -l kubernetes.io/arch=ppc64le | grep worker

2. Taint the Power Customers

oc adm taint nodes node1 kubernetes.io/arch=ppc64le:NoSchedule

Also note, the taints are flipped (intel is tained with power taint)

3. Edit manifests/overlays/multi-taint-front-end/09-front-end-dep.yaml
4. Update the toleration to match your architecture taint.
5. Save the file
6. Run the oc apply command oc apply -f manifests/overlays/multi-taint-front-end/09-front-end-dep.yaml
7. Check the location of the workers

❯ oc get pods -o wide -l name=front-end                                      
NAME                         READY   STATUS    RESTARTS   AGE    IP            NODE                                   NOMINATED NODE   READINESS GATES
front-end-69c64bf86f-98nkc   0/1     Running   0          9s     10.128.3.99   rdr-mac-cust-el-tmwmg-worker-2-5hbxg   <none>           <none>
front-end-7f4f4844c8-x79zn   1/1     Running   0          103s   10.130.2.33   mac-01a7-worker-1                      <none>           <none>

You might have to give a few minutes before the workload shifts.

8. Check again to see the workload is moved to an intel node:

❯ oc get pods -o wide -l name=front-end
NAME                         READY   STATUS    RESTARTS   AGE   IP            NODE                                   NOMINATED NODE   READINESS GATES
front-end-69c64bf86f-98nkc   1/1     Running   0          35s   10.128.3.99   rdr-mac-cust-el-tmwmg-worker-2-5hbxg   <none>           <none>

Ref: OpenShift 4.14: Understanding taints and tolerations

Summary

These are different techniques to help schedule/control workload placement and help you explore Multi-Arch Compute.

My colleague, Punith, and I have also posted two documents on further controlling workload placement:

1. Multi-Arch Compute: Node Selector

• https://community.ibm.com/community/user/powerdeveloper/blogs/paul-bastide/2024/01/09/multi-arch-compute-node-selector

2. Controlling Pod placement based on weighted node-affininty with your Multi-Arch Compute cluster

• https://community.ibm.com/community/user/powerdeveloper/blogs/punith-kenchappa/2024/02/27/controlling-pod-placement-based-on-node-affinity?CommunityKey=daf9dca2-95e4-4b2c-8722-03cd2275ab63