With the third new release this year, Red HatOpenShift4.17 is now generally available including for IBM® Power®. You can read the release notes here and find the guide for installing OpenShift 4.17 on Power here. This release builds on features included in Red Hat OpenShift 4.15 and 4.16, including an important update to multi-architecture compute that helps clients automate their modernization journeys with Power. Other updates and enhancements for clients deploying on Power focus on scalability, resource optimization, security, developer and system administrator productivity, and more. Here is an overview of key new features and improvements specifically relevant to Power:
Included with Red Hat OpenShift 4.17 is an update to multi-architecture compute called the Multiarch Tuning Operator. The Multiarch Tuning Operator optimizes workload management across different architectures such as IBM Power, IBM Z, and x86, including single-architecture clusters transitioning to multi-architecture environments. It allows systems administrators to handle scheduling and resource allocation across these different architectures by ensuring workloads are correctly directed to the nodes of compatible architectures. The Multiarch Tuning Operator in OpenShift 4.17 further helps clients optimize resource allocation with policies that automatically place workloads on the most appropriate architecture. This also improves system administrator productivity and is especially useful with business-critical workloads that require high performance or need specific architecture capabilities, such as data-intensive applications often found running on Power.
IPI with FIPS mode creates certificates that are FIPS compliant and makes sure the Nodes/Operators are using the proper cryptographic profiles.
Confirm your host is in FIPS Mode and a RHEL9 equivalent stream.
fips-mode-setup --check
Note, you must reboot after enabling fips or this binary will not function.
Download the oc
# curl -O https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.18.0-ec.2/openshift-client-linux-ppc64le-rhel9.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 44 32.4M 44 14.3M 0 0 14.1M 0 0:00:02 0:00:01 0:00:01 1100 32.4M 100 32.4M 0 0 17.0M 0 0:00:01 0:00:01 --:--:-- 17.0M
Extract the binary files
# tar xvf openshift-client-linux-ppc64le-rhel9.tar.gz
oc
kubectl
README.md
You can optionally move the oc and kubectl files to /usr/local/bin/
Download the ccoctl
# curl -O https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.18.0-ec.2/ccoctl-linux-rhel9.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 44 32.4M 44 14.3M 0 0 14.1M 0 0:00:02 0:00:01 0:00:01 1100 32.4M 100 32.4M 0 0 17.0M 0 0:00:01 0:00:01 --:--:-- 17.0M
Extract the ccoctl binary file
# tar xvf ccoctl-linux-rhel9.tar.gz ccoctl
ccoctl
Change the permissions to make ccoctl executable by running the following command:
# chmod 755 ccoctl
Copy over your pull-secret.txt
Get the Credentials Request pull spec from the release image https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.18.0-ec.2/release.txt
Note, with a FIPS host, you’ll want to use rhel9 as it supports FIPS https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/4.18.0-ec.2/openshift-client-linux-ppc64le-rhel9.tar.gz
Unarchive openshift-install-rhel9-ppc64le.tar.gz
Create the install-config.yaml using openshift-install-fips create install-config per https://developer.ibm.com/tutorials/awb-deploy-ocp-on-power-vs-ipi/
Edit install-config.yaml and add a new line at the end fips: true
Create the manifests openshift-install-fips create manifests
# openshift-install-fips create manifests
WARNING Release Image Architecture not detected. Release Image Architecture is unknown
INFO Consuming Install Config from target directory
INFO Adding clusters...
INFO Manifests created in: cluster-api, manifests and openshift
Copy the cred reqs into the right folder and confirm they are present
Create the cluster BASE_DOMAIN=powervs-openshift-ipi.cis.ibm.net RELEASE_ARCHITECTURE="ppc64le" openshift-install-fips create cluster
INFO Creating infrastructure resources...
INFO Started local control plane with envtest
INFO Stored kubeconfig for envtest in: /root/install/t/20241031c/.clusterapi_output/envtest.kubeconfig
INFO Running process: Cluster API with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:45201 --webhook-port=40159 --webhook-cert-dir=/tmp/envtest-serving-certs-1721884268 --kubeconfig=/root/install/t/20241031c/.clusterapi_output/envtest.kubeconfig]
INFO Running process: ibmcloud infrastructure provider with args [--provider-id-fmt=v2 --v=5 --health-addr=127.0.0.1:37207 --webhook-port=35963 --webhook-cert-dir=/tmp/envtest-serving-certs-3500602992 --kubeconfig=/root/install/t/20241031c/.clusterapi_output/envtest.kubeconfig]
INFO Creating infra manifests...
INFO Created manifest *v1.Namespace, namespace= name=openshift-cluster-api-guests
INFO Created manifest *v1beta1.Cluster, namespace=openshift-cluster-api-guests name=fips-fd4f6
INFO Created manifest *v1beta2.IBMPowerVSCluster, namespace=openshift-cluster-api-guests name=fips-fd4f6
INFO Created manifest *v1beta2.IBMPowerVSImage, namespace=openshift-cluster-api-guests name=rhcos-fips-fd4f6
INFO Done creating infra manifests
INFO Creating kubeconfig entry for capi cluster fips-fd4f6
INFO Waiting up to 30m0s (until 9:06AM EDT) for network infrastructure to become ready...
INFO Network infrastructure is ready
INFO Created manifest *v1beta2.IBMPowerVSMachine, namespace=openshift-cluster-api-guests name=fips-fd4f6-bootstrap
INFO Created manifest *v1beta2.IBMPowerVSMachine, namespace=openshift-cluster-api-guests name=fips-fd4f6-master-0
INFO Created manifest *v1beta2.IBMPowerVSMachine, namespace=openshift-cluster-api-guests name=fips-fd4f6-master-1
INFO Created manifest *v1beta2.IBMPowerVSMachine, namespace=openshift-cluster-api-guests name=fips-fd4f6-master-2
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=fips-fd4f6-bootstrap
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=fips-fd4f6-master-0
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=fips-fd4f6-master-1
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=fips-fd4f6-master-2
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=fips-fd4f6-bootstrap
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=fips-fd4f6-master
INFO Waiting up to 15m0s (until 9:02AM EDT) for machines [fips-fd4f6-bootstrap fips-fd4f6-master-0 fips-fd4f6-master-1 fips-fd4f6-master-2] to provision...
INFO Control-plane machines are ready
INFO Cluster API resources have been created. Waiting for cluster to become ready...
INFO Consuming Cluster API Manifests from target directory
INFO Consuming Cluster API Machine Manifests from target directory
INFO Waiting up to 20m0s (until 9:21AM EDT) for the Kubernetes API at https://api.fips.powervs-openshift-ipi.cis.ibm.net:6443...
INFO API v1.31.1 up
INFO Waiting up to 45m0s (until 9:47AM EDT) for bootstrapping to complete...
INFO Destroying the bootstrap resources...
INFO Waiting up to 5m0s for bootstrap machine deletion openshift-cluster-api-guests/fips-fd4f6-bootstrap...
INFO Shutting down local Cluster API controllers...
INFO Stopped controller: Cluster API
INFO Stopped controller: ibmcloud infrastructure provider
INFO Shutting down local Cluster API control plane...
INFO Local Cluster API system has completed operations
INFO no post-destroy requirements for the powervs provider
INFO Finished destroying bootstrap resources
INFO Waiting up to 40m0s (until 10:16AM EDT) for the cluster at https://api.fips.powervs-openshift-ipi.cis.ibm.net:6443 to initialize...
If you have any doubts, you can start a second terminal session and use the kubeconfig to verify access:
# oc --kubeconfig=auth/kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
fips-fd4f6-master-0 Ready control-plane,master 41m v1.31.1
fips-fd4f6-master-1 Ready control-plane,master 41m v1.31.1
fips-fd4f6-master-2 Ready control-plane,master 41m v1.31.1
fips-fd4f6-worker-srwf2 Ready worker 7m37s v1.31.1
fips-fd4f6-worker-tc28p Ready worker 7m13s v1.31.1
fips-fd4f6-worker-vrlrq Ready worker 7m12s v1.31.1
You can also check oc --kubeconfig=auth/kubeconfig get co
19. When it’s complete you can login and use your fips enabled cluster
The Linux Pressure Stall Information, as part of the Control Group v2, provides an accurate accounting of a containers cpu, memory and io. The psi stats allow accurate and limited access to resources – no over-committing and no over-sizing.
However, it sometimes is difficult to see if the a container is being limited and could use more resources assigned.
This article is designed to help you diagnose and check your pods so you can get the best out of your workloads.
Check your workload
You can check the container in your Pod’s cpu.stat:
Find the containerId
[root@cpi-c7b2-bastion-0 ~]# oc get pod -n test test-pod -oyaml | grep -i containerID
- containerID: cri-o://c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea
This will show you all the pods that are under pressure.
for PRESSURE in $( find /sys/fs/cgroup/kubepods.slice/ -iname io.pressure)
do
if [ ! -z "$(cat ${PRESSURE} | grep -v total=0)" ]
then
if [ ! -z "$(cat ${PRESSURE} | grep -v "avg10=0.00 avg60=0.00 avg300=0.00")" ]
then
echo ${PRESSURE}
fi
fi
done
❯ cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde03ef16_000a_4198_9e04_ac96d0ea33c5.slice/crio-d200161683a680588c4de8346ff58d633201eae2ffd558c8d707c4836215645e.scope/io.pressure
some avg10=14.02 avg60=14.16 avg300=13.99 total=4121355556
full avg10=14.02 avg60=14.16 avg300=13.99 total=4121050788
In this case, I was able to go in and icnrease the total IO.
Tweak
You can tweak the cpu.pressure settings temporarily for a pod or system so the time used to evaluate is extended (this is the longest time possible).
The maximum window size is 10 seconds, and if you have kernel version less than 6.5 then the minimum window size is 500ms.
cat << EOF > /sys/fs/cgroup/cpu.pressure
some 10000000 10000000
full 10000000 10000000
EOF
Disabling psi in OpenShift
There are two methods to disable psi in OpenShift, the first is to set a kernel parameter, and the second is to switch from cgroupsv2 to cgroups.
I found a cool article on Cert Manager with IPI PowerVS
Simplify certificate management on OpenShift across multiple architectures
Chirag Kyal is a Software Engineer at Red Hat… has authored an article about deploying IPI PowerVS and Cert Manager on IBM Cloud.
Check out the article about efficient certificate management techniques on Red Hat OpenShift using the cert-manager Operator for OpenShift’s multi-architecture support.
I’ve developed the following script to help you get started deploying multiarchitecture applications and show elaborate on the techniques for controllin multiarch compute. This script uses the sock-shop application which is available at https://github.com/ocp-power-demos/sock-shop-demo . This series of instructions for sock-shop-demo requires kustomize and following the readme.md in the repository to setup the username and password for mongodb.
You do not need to do every step that follows, please feel free to install/use what you’d like. I recommend the kustomize install with multi-no-ns, and then playing with the features you find interesting. Note, multi-no-ns requires no namespace.
The layout of the application is described in this diagram:
Deploying a non-multiarch Intel App
This deployment shows the Exec errors and pod scheduling errors that are encountered when scheduling Intel only Pods on Power.
For these steps, you are going to clone the ocp-power-demos’s sock-shop-demo and then experiment to resolve errors so the application is up and running.
The reason kustomize is used is due to the sort order feature in the binary.
Update the manifests/overlays/single/env.secret file with a username and password for mongodb. openssl rand -hex 10 is a good tip to generating a random password. You’ll need to copy this env.secret in each ‘overlays/` folder that is used in the demo.
You might be lucky enough for the scheduler to assign these to Intel only nodes.
At this point if they are all Running with no restarts, yes it’s running.
Grab the external URL
❯ oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
sock-shop sock-shop-test-user-4.apps.rdr-mac-cust-d.rdr-xyz.net front-end 8079 edge/Redirect None
Open a Browser, and navigate around. Try registering a user.
It failed for me.
Cordon Power nodes
The purpose is to cordon the Power Nodes and delete the existing pod so you get the Pod running on the architecture you want. This is only recommended on a dev/test system and on the worker nodes.
Find the Power workers
oc get nodes -l kubernetes.io/arch=ppc64le | grep worker
For each of the Power, cordon the nodes
oc adm cordon node/<worker>
List the front-end app pods
❯ oc get pods -l name=front-end
NAME READY STATUS RESTARTS AGE
front-end-648fdf6957-bjk9m 0/1 CrashLoopBackOff 13 (26s ago) 42m
Delete the front-end pods.
oc delete pod/front-end-648fdf6957-bjk9m
The app should be running correctly at this point.
Use a Node Selector for the Application
Demonstrate how to use node selector to put the workload on the right nodes.
These microservices use Deployments. We can modify the deployment to use NodeSelectors.
Edit the manifests/overlays/single/09-front-end-dep.yaml or oc edit deployment/front-end
Find the nodeSelector field and add an architecture limitation using a Node label:
With many of these applications, there are architecture specific alternatives. You can run without NodeSelectors to get the workload scheduled where there is support.
To switch to Node selectors use across Power/Intel.
Switch to oc project sock-shop
Delete the Pods and Recreate (this is a manifest-listed set of images)
❯ oc get pod -l name=rabbitmq -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rabbitmq-65c75db8db-9jqbd 2/2 Running 0 96s 10.130.2.31 mac-01a7-worker-1 <none> <none>
Process the template with the NFS_PATH and NFS_SERVER
# oc process -f storage-class-nfs-template.yaml -p NFS_PATH=/data -p NFS_SERVER=10.17.2.138 | oc apply -f –
deployment.apps/nfs-client-provisioner created
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
storageclass.storage.k8s.io/nfs-client created
