This is a recommended article on oc-mirror and getting started with a fundamental tool in OpenShift.
This guide demonstrates the use of
oc-mirrorv2 to assist in populating a local Red Hat Quay registry that will be used for a disconnected installation, and includes the steps used to configureopenshift-marketplaceto use catalog sources that point to the local Red Hat Quay registry.
The Linux Pressure Stall Information, as part of the Control Group v2, provides an accurate accounting of a containers cpu, memory and io. The psi stats allow accurate and limited access to resources – no over-committing and no over-sizing.
However, it sometimes is difficult to see if the a container is being limited and could use more resources assigned.
This article is designed to help you diagnose and check your pods so you can get the best out of your workloads.
You can check the container in your Pod’s cpu.stat:
[root@cpi-c7b2-bastion-0 ~]# oc get pod -n test test-pod -oyaml | grep -i containerID
- containerID: cri-o://c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea
[root@cpi-c7b2-bastion-0 ~]# oc rsh -n test test-pod
sh-4.4# find /sys -iname '*c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea*'
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4b90d9_20f9_427d_9414_9964f32379dc.slice/crio-c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea.scope
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4b90d9_20f9_427d_9414_9964f32379dc.slice/crio-conmon-c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea.scope
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4b90d9_20f9_427d_9414_9964f32379dc.slice/crio-conmon-c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea.scope/cpu.stat
usage_usec 11628232854
user_usec 8689145332
system_usec 2939087521
core_sched.force_idle_usec 0
nr_periods 340955
nr_throttled 8
throttled_usec 8012
nr_bursts 0
burst_usec 0
nr_throttled and throttled_usec. This is really a minor impact for a container.nr_throttled 8
throttled_usec 8012
If the container had a higher number of throttled events, you want to check the number of cpus or memory that your container is limited to, such as:
nr_throttled 103
throttled_usec 22929315
❯ NS=test
❯ POD=test-pod
❯ oc get -n ${NS} pod ${POD} -ojson | jq -r '.spec.containers[].resources.limits.cpu'
8
You can check the real-time stats top for your container pressure. Log on to your host.
find /sys/fs/cgroup/kubepods.slice/ -iname cpu.pressure | xargs -t -I {} cat {} | grep -v total=0
find /sys/fs/cgroup/kubepods.slice/ -iname memory.pressure | xargs -t -I {} cat {} | grep -v total=0
find /sys/fs/cgroup/kubepods.slice/ -iname io.pressure | xargs -t -I {} cat {} | grep -v total=0
This will show you all the pods that are under pressure.
for PRESSURE in $( find /sys/fs/cgroup/kubepods.slice/ -iname io.pressure)
do
if [ ! -z "$(cat ${PRESSURE} | grep -v total=0)" ]
then
if [ ! -z "$(cat ${PRESSURE} | grep -v "avg10=0.00 avg60=0.00 avg300=0.00")" ]
then
echo ${PRESSURE}
fi
fi
done
❯ cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde03ef16_000a_4198_9e04_ac96d0ea33c5.slice/crio-d200161683a680588c4de8346ff58d633201eae2ffd558c8d707c4836215645e.scope/io.pressure
some avg10=14.02 avg60=14.16 avg300=13.99 total=4121355556
full avg10=14.02 avg60=14.16 avg300=13.99 total=4121050788
In this case, I was able to go in and icnrease the total IO.
You can tweak the cpu.pressure settings temporarily for a pod or system so the time used to evaluate is extended (this is the longest time possible).
The maximum window size is 10 seconds, and if you have kernel version less than 6.5 then the minimum window size is 500ms.
cat << EOF > /sys/fs/cgroup/cpu.pressure
some 10000000 10000000
full 10000000 10000000
EOF
There are two methods to disable psi in OpenShift, the first is to set a kernel parameter, and the second is to switch from cgroupsv2 to cgroups.
You can switch from cgroupsv2 to cgroups – Configuring the Linux cgroup version on your nodes.
❯ oc patch nodes.config cluster --type merge -p '{"spec": {"cgroupMode": "v1"}}'
You’ll have to wait for each of the Nodes to restart.
In OpenShift, you can disable psi in using a MachineConfig.
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 99-worker-psi-disable
spec:
kernelArguments:
- psi=0
You can check to see if it is enabled by checking one of the cpu.pressure, io.pressure or memory.pressure files. You’ll see “Operation not supported”.
sh-5.1# cat /sys/fs/cgroup/cpu.pressure
cat: /sys/fs/cgroup/cpu.pressure: Operation not supported
or
oc debug node/<node_name>
chroot /host
stat -c %T -f /sys/fs/cgroup
tmpfs
Linux PSI is pretty awesome. However, you should check your workload and verify it’s running correctly.
kernel/sched/psi.cOur colleague at Red Hat Dylan Orzel posted an article on Building multi-architecture container images on OpenShift Container Platform clusters
In this article we’ll explore how to make use of the built-in build capabilities available in Red Hat OpenShift 4 in a multi-arch compute environment, and how to make use of nodeSelectors to schedule builds on nodes of the architecture of our choosing.
Here are some things around IBM Power Systems and Red Hat OpenShift you should know about:
The IBM Power team has updated the list of containers they build with support for ppc64le. The list is kept at https://community.ibm.com/community/user/powerdeveloper/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr
The updates are:
| system-logger | v1.19.0 | podman pull icr.io/ppc64le-oss/system-logger-ppc64le:v1.14.0 | July 18, 2024 |
| postgres-operator | v15.7 | podman pull icr.io/ppc64le-oss/postgres-operator-ppc64le:v15.7 | July 18, 2024 |
| postgresql | v14.12.0-bv | podman pull icr.io/ppc64le-oss/postgresql:v14.12.0-bv | July 9, 2024 |
| mongodb | 5.0.26 | podman pull icr.io/ppc64le-oss/mongodb-ppc64le:5.0.26 | April 9, 2024 |
| 6.0.13 | podman pull icr.io/ppc64le-oss/mongodb-ppc64le:6.0.13 | April 9, 2024 |
Trivy and Starboard are now available per https://community.ibm.com/community/user/powerdeveloper/blogs/gerrit-huizenga/2024/07/17/aqua-trivy-and-starboard-for-scanning-gitlab-on-ib
You can download the Trivy RPM using:
rpm -ivh https://github.com/aquasecurity/trivy/releases/download/v0.19.2/trivy_0.19.2_Linux-PPC64LE.rpmOr you could use Starboard directly from https://github.com/aquasecurity/trivy-operator/releases/tag/v0.22.0
These provide some nice security features and tools for IBM Power containers.
The OpenShift Routes project supports automatically getting a certificate for OpenShift routes from any cert-manager Issuer, similar to annotating an Ingress or Gateway resource in vanilla Kubernetes.
You can download the helm chart from https://github.com/cert-manager/openshift-routes/releases
Or you can use:
helm install openshift-routes -n cert-manager oci://ghcr.io/cert-manager/charts/openshift-routes
OpenBao exists to provide a software solution to manage, store, and distribute sensitive data including secrets, certificates, and keys.
OpenBAO has released v2.0.0
You can use helm to install on IBM Power use the values.openshift.yaml link
helm repo add openbao https://openbao.github.io/openbao-helm
helm install openbao openbao/openbaoThe Containers are at https://quay.io/repository/openbao/openbao?tab=tags&tag=latest
Red Hat OpenShift 4.16 is generally available for upgrades and new installations, and as of today is announced. It is based on Kubernetes 1.29 with the CRI-O 1.29 runtime, RHEL CoreOS 9.4. You can read the release notes at https://docs.openshift.com/container-platform/4.16/release_notes/ocp-4-16-release-notes.html
Some cool features you can use are:
– oc adm upgrade status command, which decouples status information from the existing oc adm upgrade command and provides specific information regarding a cluster update, including the status of the control plane and worker node updates. https://docs.openshift.com/container-platform/4.16/updating/updating_a_cluster/updating-cluster-cli.html#update-upgrading-oc-adm-upgrade-status_updating-cluster-cli
– Tech Preview and Generally Available Table – https://docs.openshift.com/container-platform/4.16/release_notes/ocp-4-16-release-notes.html#ocp-4-16-technology-preview-tables_release-notes
FYI: google/go-containerregistry has a new release v0.19.2. This adds a new feature we care about:
crane mutate myimage --set-platform linux/arm64
This release also supports using podman’s authfile from the REGISTRY_AUTH_FILE file.
I found a cool article on Cert Manager with IPI PowerVS
Simplify certificate management on OpenShift across multiple architectures
Chirag Kyal is a Software Engineer at Red Hat… has authored an article about deploying IPI PowerVS and Cert Manager on IBM Cloud.
Check out the article about efficient certificate management techniques on Red Hat OpenShift using the cert-manager Operator for OpenShift’s multi-architecture support.
The new information for the end of April is:
The IBM Linux on Power team released more images to their IBM Container Registry (ICR) here are the new ones:
| milvus | v2.3.3 | docker pull icr.io/ppc64le-oss/milvus-ppc64le:v2.3.3 | April 2, 2024 |
| rust | 1.66.1 | docker pull icr.io/ppc64le-oss/rust-ppc64le:1.66.1 | April 2, 2024 |
| opensearch | 2.12.0 | docker pull icr.io/ppc64le-oss/opensearch-ppc64le:2.12.0 | April 16, 2024 |
Original post was at https://community.ibm.com/community/user/powerdeveloper/blogs/paul-bastide/2024/04/26/getting-started-with-a-sock-shop-a-sample-multi-ar?CommunityKey=daf9dca2-95e4-4b2c-8722-03cd2275ab63
I’ve developed the following script to help you get started deploying multiarchitecture applications and show elaborate on the techniques for controllin multiarch compute. This script uses the sock-shop application which is available at https://github.com/ocp-power-demos/sock-shop-demo . This series of instructions for sock-shop-demo requires kustomize and following the readme.md in the repository to setup the username and password for mongodb.
You do not need to do every step that follows, please feel free to install/use what you’d like. I recommend the kustomize install with multi-no-ns, and then playing with the features you find interesting. Note, multi-no-ns requires no namespace.
The layout of the application is described in this diagram:
This deployment shows the Exec errors and pod scheduling errors that are encountered when scheduling Intel only Pods on Power.
For these steps, you are going to clone the ocp-power-demos’s sock-shop-demo and then experiment to resolve errors so the application is up and running.
I’d recommend running this from a bastion.
git clone https://github.com/ocp-power-demos/sock-shop-demo
sock-shop-demo folderkustomize – this tool enable a ordered layout of the resources. You’ll also need oc installed.curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
Ref: https://kubectl.docs.kubernetes.io/installation/kustomize/binaries/
The reason kustomize is used is due to the sort order feature in the binary.
manifests/overlays/single/env.secret file with a username and password for mongodb. openssl rand -hex 10 is a good tip to generating a random password. You’ll need to copy this env.secret in each ‘overlays/` folder that is used in the demo.❯ kustomize build manifests/overlays/single | oc apply -f -
This create a full application within the OpenShift project.
To see the layout of the application you can see the third diagram of the layout (except these are only Intel images) https://github.com/ocp-power-demos/sock-shop-demo/blob/main/README.md#diagrams
oc get pods -owide❯ oc get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
carts-585dc6c878-wq6jg 0/1 Error 6 (2m56s ago) 6m21s 10.129.2.24 mac-01a7-worker-0 <none> <none>
carts-db-78f756b87c-r4pl9 1/1 Running 0 6m19s 10.131.0.32 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
catalogue-77d7c444bb-wnltt 0/1 CrashLoopBackOff 6 (8s ago) 6m17s 10.130.2.21 mac-01a7-worker-1 <none> <none>
catalogue-db-5bc97c6b98-v9rdp 1/1 Running 0 6m16s 10.131.0.33 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
front-end-648fdf6957-bjk9m 0/1 CrashLoopBackOff 5 (2m44s ago) 6m14s 10.129.2.25 mac-01a7-worker-0 <none> <none>
orders-5dbf8994df-whb9r 0/1 CrashLoopBackOff 5 (2m47s ago) 6m13s 10.130.2.22 mac-01a7-worker-1 <none> <none>
orders-db-7544dc7fd9-w9zh7 1/1 Running 0 6m11s 10.128.3.83 rdr-mac-cust-el-tmwmg-worker-2-5hbxg <none> <none>
payment-6cdff467b9-n2dql 0/1 Error 6 (2m53s ago) 6m10s 10.130.2.23 mac-01a7-worker-1 <none> <none>
queue-master-c9dcf8f87-c8drl 0/1 CrashLoopBackOff 5 (2m41s ago) 6m8s 10.129.2.26 mac-01a7-worker-0 <none> <none>
rabbitmq-54689956b9-rt5fb 2/2 Running 0 6m7s 10.131.0.34 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
session-db-7d4cc56465-dcx9f 1/1 Running 0 6m5s 10.130.2.24 mac-01a7-worker-1 <none> <none>
shipping-5ff5f44465-tbjv7 0/1 Error 6 (2m51s ago) 6m4s 10.130.2.25 mac-01a7-worker-1 <none> <none>
user-64dd65b5b7-49cbd 0/1 CrashLoopBackOff 5 (2m25s ago) 6m3s 10.129.2.27 mac-01a7-worker-0 <none> <none>
user-db-7f864c9f5f-jchf6 1/1 Running 0 6m1s 10.131.0.35 rdr-mac-cust-el-tmwmg-worker-1-6g97b <none> <none>
You might be lucky enough for the scheduler to assign these to Intel only nodes.
At this point if they are all Running with no restarts, yes it’s running.
❯ oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
sock-shop sock-shop-test-user-4.apps.rdr-mac-cust-d.rdr-xyz.net front-end 8079 edge/Redirect None
It failed for me.
The purpose is to cordon the Power Nodes and delete the existing pod so you get the Pod running on the architecture you want. This is only recommended on a dev/test system and on the worker nodes.
oc get nodes -l kubernetes.io/arch=ppc64le | grep worker
oc adm cordon node/<worker>
❯ oc get pods -l name=front-end
NAME READY STATUS RESTARTS AGE
front-end-648fdf6957-bjk9m 0/1 CrashLoopBackOff 13 (26s ago) 42m
front-end pods.oc delete pod/front-end-648fdf6957-bjk9m
The app should be running correctly at this point.
Demonstrate how to use node selector to put the workload on the right nodes.
These microservices use Deployments. We can modify the deployment to use NodeSelectors.
manifests/overlays/single/09-front-end-dep.yaml or oc edit deployment/front-endnodeSelector field and add an architecture limitation using a Node label:nodeSelector:
node.openshift.io/os_id: rhcos
kubernetes.io/arch: amd64
oc apply -f manifests/overlays/single/09-front-end-dep.yaml❯ oc get pods -l name=front-end
NAME READY STATUS RESTARTS AGE
front-end-648fdf6957-bjk9m 0/1 CrashLoopBackOff 14 (2m49s ago) 50m
front-end-7bd476764-t974g 0/1 ContainerCreating 0 40s
front-end pod on the power node.oc delete pod/front-end-648fdf6957-bjk9m
Note, you can run the following to run with nodeSelectors.
❯ kustomize build manifests/overlays/single-node-selector | oc delete -f -
❯ kustomize build manifests/overlays/single-node-selector | oc apply -f -
Are the pods running on the Intel node?
With the nodeSelector now started, you can uncordon the Power nodes. This is only recommended on a dev/test system and on the worker nodes.
oc get nodes -l kubernetes.io/arch=ppc64le | grep worker
oc adm uncordon node/<worker>
❯ oc get pods -l name=front-end
NAME READY STATUS RESTARTS AGE
front-end-6944957cd6-qmhhg 1/1 Running 0 19s
The application should be running. If not, please use:
❯ kustomize build manifests/overlays/single-node-selector | oc delete -f -
❯ kustomize build manifests/overlays/single-node-selector | oc apply -f -
The workload should be all on the intel side.
With many of these applications, there are architecture specific alternatives. You can run without NodeSelectors to get the workload scheduled where there is support.
To switch to Node selectors use across Power/Intel.
oc project sock-shop❯ kustomize build manifests/overlays/multi-no-ns | oc apply -f -
❯ oc get pods -owide
We’re going to move one of the applications’ dependencies using rabbitmq. The IBM team has created a port of Redis to ppc64le. link
image: kbudde/rabbitmq-exporter on line 32. icr.io/ppc64le-oss/rabbitmq-exporter-ppc64le:1.0.0-RC19kubernetes.io/arch: amd64 limitation on line 39kustomize build manifests/overlays/multi-no-ns | oc apply -f -❯ oc get pod -l name=rabbitmq -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rabbitmq-65c75db8db-9jqbd 2/2 Running 0 96s 10.130.2.31 mac-01a7-worker-1 <none> <none>
The pod should now start on the Power node.
You’ve taken advantage of the containers, and you can take advantage of other OpenSource container images https://community.ibm.com/community/user/powerdeveloper/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr
Taints and Tolerations provide a way to
oc get nodes -l kubernetes.io/arch=ppc64le | grep worker
oc adm taint nodes node1 kubernetes.io/arch=ppc64le:NoSchedule
Also note, the taints are flipped (intel is tained with power taint)
manifests/overlays/multi-taint-front-end/09-front-end-dep.yamloc apply -f manifests/overlays/multi-taint-front-end/09-front-end-dep.yaml❯ oc get pods -o wide -l name=front-end
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
front-end-69c64bf86f-98nkc 0/1 Running 0 9s 10.128.3.99 rdr-mac-cust-el-tmwmg-worker-2-5hbxg <none> <none>
front-end-7f4f4844c8-x79zn 1/1 Running 0 103s 10.130.2.33 mac-01a7-worker-1 <none> <none>
You might have to give a few minutes before the workload shifts.
❯ oc get pods -o wide -l name=front-end
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
front-end-69c64bf86f-98nkc 1/1 Running 0 35s 10.128.3.99 rdr-mac-cust-el-tmwmg-worker-2-5hbxg <none> <none>
Ref: OpenShift 4.14: Understanding taints and tolerations
These are different techniques to help schedule/control workload placement and help you explore Multi-Arch Compute.
My colleague, Punith, and I have also posted two documents on further controlling workload placement:
Here are my notes for setting up the SIG’s nfs-provisioner. You should follow these directions to setup the nfs-provisioner kubernetes-sigs/nfs-subdir-external-provisioner.
a. Create the namespace
oc new-project nfs-provisionerb. Annotate the namespace with elevated privileges so we can create NFS mounts
# oc label namespace/nfs-provisioner security.openshift.io/scc.podSecurityLabelSync=false --overwrite=true
namespace/nfs-provisioner labeled# oc label namespace/nfs-provisioner pod-security.kubernetes.io/enforce=privileged --overwrite=true
namespace/nfs-provisioner labeled# oc label namespace/nfs-provisioner pod-security.kubernetes.io/enforce-version=v1.24 --overwrite=true
namespace/nfs-provisioner labeled# oc label namespace/nfs-provisioner pod- security.kubernetes.io/audit=privileged --overwrite=true
namespace/nfs-provisioner labeled# oc label namespace/nfs-provisioner pod-security.kubernetes.io/warn=privileged --overwrite=true
namespace/nfs-provisioner labeled# curl -O -L https://github.com/IBM/ocp4-power-workload-tools/manifests/storage/storage-class-nfs-template.yamloc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:nfs-provisioner:nfs-client-provisioner# oc process -f storage-class-nfs-template.yaml -p NFS_PATH=/data -p NFS_SERVER=10.17.2.138 | oc apply -f –
deployment.apps/nfs-client-provisioner created
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
storageclass.storage.k8s.io/nfs-client createdoc get podsNAME READY STATUS RESTARTS AGEnfs-client-provisioner-b8764c6bb-mjnq9 1/1 Running 0 36s❯ oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-client k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 3m27s
If you see more than the nfs-client listed, you may have to change the defaults.
oc patch storageclass storageclass-name -p ‘{“metadata”: {“annotations”: {“storageclass.kubernetes.io/is-default-class”: “false”}}}’