I was helping a colleague grab the latest ignition files for his OpenShift Container Platform workers.

1. Connect to the bastion

❯ ssh root@<hostname>

2. List the master nodes and select a master node

❯ oc get nodes -lnode-role.kubernetes.io/master=
NAME                                                        STATUS   ROLES                  AGE     VERSION
master-0.ocp-power.xyz   Ready    control-plane,master   5d19h   v1.25.2+7dab57f
master-1.ocp-power.xyz   Ready    control-plane,master   5d19h   v1.25.2+7dab57f
master-2.ocp-power.xyz   Ready    control-plane,master   5d19h   v1.25.2+7dab57f

3. Get the IP address

❯ oc get nodes master-2.ocp-power.xyz -o json | jq -r .status.addresses
[
  {
    "address": "192.168.100.240",
    "type": "InternalIP"
  },
  {
    "address": "master-2.ocp-power.xyz",
    "type": "Hostname"
  }
]

4. The Machine Config Server has the latest igntion file

❯ curl https://192.168.100.240:22623/config/worker -k -H "Accept: application/vnd.coreos.ignition+json;version=3.2.0" | jq -r . > worker-$(date +%Y-%m-%d).ign

worker can be replaced with master or any other MachineConfigPool

The machine-config-server is hosted on each of the master nodes and the bootstrap node.

Note, this makes it download ignition version 3.2.0.

5. Download the ignition file

❯ scp root@<hostname>:'~'/worker-$(date +%Y-%m-%d).ign .
worker-2023-03-02.ign                          100%  357KB 249.3KB/s   00:01    

You can use this file for your work with worker ignition.

Refer to https://github.com/openshift/machine-config-operator/blob/e39fea2d5c1d6991df4f7dd526b6292238f2ecfc/pkg/server/api.go#L196

During the week, I accumulate a lot of links/information that I want to come back to over and over again. I find it helpful to blog about them. Here are some of things I found interesting this week, and frankly helpful.

Build multi-arch images on GitHub Actions with Buildx 

If you have ever wondered how to build multi-arch containers to run on ppc64le, x86, ARM, and/or s390x with GitHub Actions, then this article is for you. GitHub Actions is a continuous integration … 

https://community.ibm.com/community/user/powerdeveloper/blogs/siddhesh-ghadi/2023/02/08/build-multi-arch-images-on-github-actions-with-bui

A comprehensive article on using docker buildx to generate manifest listed builds.

Routing Via Host with OVNKuberenetes 

OVN-Kuberenetes is the default network provider for OpenShift Container Platform. OVN-Kubernetes is based on Open Virtual Network (OVN) and provides an overlay-based networking implementation. OVN-Kubernetes … 

https://community.ibm.com/community/user/powerdeveloper/blogs/mick-tarsel/2023/01/26/routingviahost-with-ovnkuberenetes

The background and why we use OVN-Kubernetes is superb. Mick did a bang up job on it.

Making a OpenShift Container Platform StorageClass the default

I had to recollect how to set the StorageClass as the default. You can check the current storageclass setting

❯ oc get storageclass
NAME                   PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client             k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  18m

Patch the storageclass name – nfs-client

❯ oc patch storageclass nfs-client -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

Check the storage class is now the default

❯ oc get storageclass
NAME                   PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  18m

References

1. Changing the default storage class

The above is a fast way to set the StorageClass default.

Setting Namespace Security Labels

In case you need to set it for a specific namespace such as nfs-provisioner.

oc label namespace/nfs-provisioner security.openshift.io/scc.podSecurityLabelSync=false --overwrite=true
oc label namespace/nfs-provisioner pod-security.kubernetes.io/enforce=privileged --overwrite=true
oc label namespace/nfs-provisioner pod-security.kubernetes.io/audit=privileged --overwrite=true
oc label namespace/nfs-provisioner pod-security.kubernetes.io/warn=privileged --overwrite=true

The above is a quick macro for setting it for a given namespace so you can get privileged access.

My teammate was investigating an SSHD config change and hit a stuck MachineConfigPool. Here are some steps we followed to get it unstuck.

Steps

1. Verify that the MachineConfigPool is stuck updating

❯ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-0de63bfa1c0db0777031adddb3286fbc   False     True       True       3              0                   0                     3                      9d
worker   rendered-worker-38e4049eaf0b7fca848408378092e607   True      False      False      3              3                   3                     0                      9d

2. Find out for one of your nodes in the mcp that is stuck (for instance, master-0)

❯ oc get pods -n openshift-machine-config-operator --field-selector spec.nodeName=master-0
NAME                          READY   STATUS    RESTARTS   AGE
machine-config-daemon-t8x8j   2/2     Running   2          35h
machine-config-server-kfx8n   1/1     Running   1          35h

3. Check the logs and grab the rendered-master

❯ oc logs pod/machine-config-daemon-tgnss -n openshift-machine-config-operator
...
E0124 07:19:26.746977  780508 on_disk_validation.go:208] content mismatch for file "/etc/ssh/sshd_config" (-want +got):
  bytes.Join({
-       "\n#\t",
+       "#       ",
        "$OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $\n\n# Th",
        "is is the sshd server system-wide configuration file.  See\n# ssh",
        ... // 1437 identical bytes
        "keys and .ssh/authorized_keys2\n# but this is overridden so insta",
        "llations will only check .ssh/authorized_keys\nAuthorizedKeysFile",
-       `       `,
+       "      ",
        ".ssh/authorized_keys\n\n#AuthorizedPrincipalsFile none\n\n#Authorize",
        "dKeysCommand none\n#AuthorizedKeysCommandUser nobody\n\n# For this ",
        ... // 2258 identical bytes
        "E LC_MEASUREMENT\nAcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE\nAcc",
...
+       "\n",
  }, "")
E0124 07:19:26.747042  780508 writer.go:200] Marking Degraded due to: unexpected on-disk state validating against rendered-master-0de63bfa1c0db0777031adddb3286fbc: content mismatch for file "/etc/ssh/sshd_config"
I0124 07:19:28.973484  780508 daemon.go:1248] Current+desired config: rendered-master-0de63bfa1c0db0777031adddb3286fbc
...

4. OK, this looks like a problem with the whitespace, and inspect the URL decoded version’s whites pace vim :set list

> oc get mc rendered-master-0de63bfa1c0db0777031adddb3286fbc -o yaml > out.yaml

You may have to update the white space.

5. Check the reasons for the failure if the whitespace doesn’t fix it.

> oc describe mcp master

Message:
    Node master-0 is reporting: 
        "unexpected on-disk state validating against rendered-master-0de63bfa1c0db0777031adddb3286fbc: 
        mode mismatch for file: \"/etc/ssh/sshd_config\"; 
        expected: -rw-------/384/0600; received: -rw-r--r--/420/0644", 
        Node master-1 is reporting: "unexpected on-disk state validating 
        against rendered-master-0de63bfa1c0db0777031adddb3286fbc: content 
        mismatch for file \"/etc/ssh/sshd_config\"", Node master-2 is reporting:
        "unexpected on-disk state validating against 
        rendered-master-0de63bfa1c0db0777031adddb3286fbc: content mismatch for file 
        \"/etc/ssh/sshd_config\""

In this case, the local files were edited while preparing the ideal sshd_config and needed a forced update.

6. Force the machine-config to refresh files.

> touch /run/machine-config-daemon-force

7. You should see the states change after the node reboots.

Events:
  Type    Reason            Age    From                                    Message
  ----    ------            ----   ----                                    -------
  Normal  AnnotationChange  5m19s  machineconfigcontroller-nodecontroller  Node master-0 now has machineconfiguration.openshift.io/state=Done

  degradedMachineCount: 2
  machineCount: 3
  observedGeneration: 500
  readyMachineCount: 0
  unavailableMachineCount: 2
  updatedMachineCount: 0

If you need to select a file from the rendered config:

> oc get mc rendered-master-0de63bfa1c0db0777031adddb3286fbc -o yaml | yq -r '.spec.config[].files[] | select(.path == "/etc/ssh/sshd_config").contents.source'
data:,%0A%23%09$OpenBSD:%20sshd_config%2Cv%201.103
...

References

• Red Hat KnowledgeBase: How to skip validation of failing / stuck MachineConfig in OCP 4?
• How to create a file on all openshift container platform 4.0 nodes
• Disable SSH weak ciphers/algorithms on OpenShift Container Platform 4

To demonstrate a multi-tiered web application, I used ghost, the microblogging platform to deploy the application using kustomize. Kustomize is a higher-level orchestration of the steps to deploy an application with environment specific overlays.

git clone https://github.com/prb112/openshift-demo.git

$ brew install kustomize

$ ENV_PASS=$(openssl rand -hex 10)
$ echo ${ENV_PASS}

$ export WEB_DOMAIN=https://web-route-ghost.apps.$(oc get ingress.config.openshift.io cluster -o yaml | grep domain | awk '{print $NF}')
$ echo ${WEB_DOMAIN}

$ cat secrets/01_db_secret.yml | sed "s|ENV_PASS|${ENV_PASS}|" | oc apply -f -

$ cat secrets/02_web_cm.yml | sed "s|WEB_DOMAIN|${WEB_DOMAIN}|" | oc apply -f -

$ oc apply -k overlays/dev
namespace/ghost configured
service/db-service unchanged
service/web unchanged
persistentvolumeclaim/db-pvc unchanged
persistentvolumeclaim/web-content unchanged
deployment.apps/ghost-db unchanged
deployment.apps/web unchanged
route.route.openshift.io/web-route unchanged

$ oc delete -k overlays/dev
namespace "ghost" deleted
service "db-service" deleted
service "web" deleted
persistentvolumeclaim "db-pvc" deleted
persistentvolumeclaim "web-content" deleted
deployment.apps "ghost-db" deleted
deployment.apps "web" deleted
route.route.openshift.io "web-route" deleted

$ oc get cm -o yaml