Author: Paul

  • Handy Dandy Tricks for the Week

    As I work with Linux and the OpenShift Container Platform, I run into some handy tips, tricks and patterns that make my job easier, and I wanted to share with you all (and so I can find these details in the future):

    1. Verifying etcd data is encrypted

    The etcd depends on the Kube-APIServer and OpenShift-APIServer to encrypt a subset of the API Resources, and transparently with AES-CBC (and eventually AES-GCM). The subset includes: Secrets, Config maps, Routes, OAuth access tokens, OAuth authorize tokens.

    To verify the the contents are actually encrypted one can look at https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#verifying-that-data-is-encrypted or https://docs.openshift.com/container-platform/4.12/security/encrypting-etcd.html

    2. Using a Pod to run lsof across the whole Node

    You can create a debug Pod with lsof.

    1. Create a debug Pod with elevated privileges.
    cat << EOF > bpftrace-diagnostic-0.yaml
    kind: Pod
    apiVersion: v1
    metadata:
      annotations:
        openshift.io/scc: node-exporter
      name: bpftrace-diagnostic-0
      namespace: openshift-etcd
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      priority: 2000000000
      schedulerName: default-scheduler
      enableServiceLinks: true
      terminationGracePeriodSeconds: 30
      preemptionPolicy: PreemptLowerPriority
      containers:
        - name: diagnostic
          image: quay.io/centos/centos:stream8
          imagePullPolicy: IfNotPresent
          command: [ "sh", "-c", "sleep inf" ]
          resources:
            requests:
              cpu: 1000m
              memory: 2048Mi
          volumeMounts:
          - name: host-sys
            mountPath: /sys
          terminationMessagePath: /dev/termination-log
          securityContext:
            privileged: true
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              add:
                - CAP_SYS_ADMIN
                - CAP_FOWNER
                - NET_ADMIN
                - SYS_ADMIN
              drop:
                - ALL
            runAsUser: 0
            runAsGroup: 0
            runAsNonRoot: false
            readOnlyRootFilesystem: false
            allowPrivilegeEscalation: true
      volumes:
      - name: host-sys
        hostPath:
          path: /sys
          type: Directory
      nodeName: master-0
      priorityClassName: system-cluster-critical
      hostPID: true
      hostIPC: true
      hostNetwork: true
    EOF
    
    1. Delete an existing pod and apply a new pod.
    oc delete pod/bpftrace-diagnostic-0; oc apply -f bpftrace-diagnostic-0.yaml
    
    1. Connect to the Pod
    oc rsh pod/bpftrace-diagnostic-0
    
    1. You can install lsof and verify the files open with a specific process.
    ❯ dnf install -y lsof
    ❯ ps -ef | grep 'etcd '
    root     1236199 1235495  0 16:31 pts/1    00:00:00 grep etcd 
    root     2273604 2273591  2 Mar17 ?        02:41:43 etcd grpc-proxy start --endpoints ........
    ❯  lsof +p 2273591
    

    With lsof you can see the open files and connections for that process.

    3. Verifying FIPS usage with GO

    Red Hat’s development site has an awesome article on Is your Go application FIPS compliant?. It shows you how to confirm if your application has FIPS/uses FIPS.

    To confirm my container used FIPS, I connected to the project, grabbed the files off the container, and saw the libraries referenced.

    $ oc project myapp
    $ oc get pods
    $ oc cp app-5d69757bd6-57g9x:/usr/bin/app /tmp/app
    $ go tool nm /tmp/app | grep FIPS
    11b5a4b0 T _cgo_af9b144e6dc6_Cfunc__goboringcrypto_FIPS_mode
    138c30d8 d _g_FIPS_mode
    137c2570 d crypto/tls.defaultCipherSuitesFIPS
    137c25f0 d crypto/tls.defaultFIPSCipherSuitesTLS13
    137c2610 d crypto/tls.defaultFIPSCurvePreferences
    102007d0 t vendor/github.com/golang-fips/openssl-fips/openssl.enableBoringFIPSMode
    138c3467 d vendor/github.com/golang-fips/openssl-fips/openssl.strictFIPS
    

    Thus, you can show that it uses FIPS mode.

    4. Verifying the ignition contents from a qcow2 image

    I was playing with Single Node OpenShift, and needed to see how out of date my code and igntion file were.

    Checking this iso tells me your RHCOS is out-of-date (about 4 months):

    isoinfo -l -i ./rhcos-live.iso  | grep -i IGN
    ----------   0    0    0          262144 Dec 13 2022 [   2378 00]  IGNITION.IMG;1 
    

    I checked the qcow2 image that the iso was converted to, and the mounted it so I could extract some files:

    ❯ modprobe nbd max_part=8
    ❯ qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/rhcos-live-2023-03-21.qcow2
    ❯ fdisk /dev/nbd0 -l
    ❯ mkdir -p /mnt/tmp-f
    ❯ mount /dev/nbd0p1 /mnt/tmp-f
    ❯ find /mnt/tmp-f/
    /mnt/tmp-f/
    /mnt/tmp-f/coreos
    /mnt/tmp-f/coreos/features.json
    /mnt/tmp-f/coreos/kargs.json
    /mnt/tmp-f/coreos/miniso.dat
    /mnt/tmp-f/EFI
    /mnt/tmp-f/EFI/redhat
    /mnt/tmp-f/EFI/redhat/grub.cfg
    /mnt/tmp-f/images
    /mnt/tmp-f/images/efiboot.img
    /mnt/tmp-f/images/ignition.img
    /mnt/tmp-f/images/pxeboot
    /mnt/tmp-f/images/pxeboot/initrd.img
    /mnt/tmp-f/images/pxeboot/rootfs.img
    /mnt/tmp-f/images/pxeboot/vmlinuz
    /mnt/tmp-f/isolinux
    /mnt/tmp-f/isolinux/boot.cat
    /mnt/tmp-f/isolinux/boot.msg
    /mnt/tmp-f/isolinux/isolinux.bin
    /mnt/tmp-f/isolinux/isolinux.cfg
    /mnt/tmp-f/isolinux/ldlinux.c32
    /mnt/tmp-f/isolinux/libcom32.c32
    /mnt/tmp-f/isolinux/libutil.c32
    /mnt/tmp-f/isolinux/vesamenu.c32
    /mnt/tmp-f/zipl.prm
    

    from link I learned I could scan the ignition.img for ‘fyre’ which was part of the ssh public key.

    I then catted out the ingintion file to confirm it had the updated content.

    ❯ xzcat /mnt/tmp-f/images/ignition.img | grep myid | wc -l
    1
    

    This shows how to dig into the contents.

    5. List of Operating System Short Ids for coreos

    I need to find the os-variant for coreos, so I could use it on libvirt. Fortunately, the libosinfo package provides a tool the details on the oeprating systems wiht osinfo-query os.

    ❯ osinfo-query os | grep coreos
     fedora-coreos-next   | Fedora CoreOS  | Next    | http://fedoraproject.org/coreos/next    
     fedora-coreos-stable | Fedora CoreOS  | Stable  | http://fedoraproject.org/coreos/stable  
     fedora-coreos-testing | Fedora CoreOS | Testing | http://fedoraproject.org/coreos/testing 
    

    I then added --os-variant=fedora-coreos-next to my virt-install command.

    6. Using qemu-kvm with a port number less than 1024

    I needed to forward a privileged port, and was blocked with qemu-kvm and virt-install. I thought there was another process that might be listening on the port… so I checked…

    ❯ netstat -plutn | grep 443
    tcp        0      0 0.0.0.0:6443            0.0.0.0:*               LISTEN      1026882/qemu-kvm    
    

    It was not, and it turned out to be a known issue with qemu-kvm and selinux.

    The ports < 1024 are privileged, and only a root process (or a process with CAP_NET_BIND_SERVICE capabilities in Linux) can bind a socket to them. link to article

    I ran the following:

    ❯ setcap CAP_NET_BIND_SERVICE+ep /usr/libexec/qemu-kvm
    

    I recreated the image with virt-install and --network network=default,model=e1000,hostfwd=tcp::443-:443,hostfwd=tcp::6443-:6443'

    Good luck, it worked so far for me.

    7. Using bpftrace with a Pod

    You can use bpftrace to do analysis at the Kernel level (some kprobes are not available).

    1. Create a debug Pod with elevated privileges.
    cat << EOF > bpftrace-diagnostic-0.yaml
    kind: Pod
    apiVersion: v1
    metadata:
      annotations:
        openshift.io/scc: node-exporter
      name: bpftrace-diagnostic-0
      namespace: openshift-etcd
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      priority: 2000000000
      schedulerName: default-scheduler
      enableServiceLinks: true
      terminationGracePeriodSeconds: 30
      preemptionPolicy: PreemptLowerPriority
      containers:
        - name: diagnostic
          image: quay.io/centos/centos:stream8
          imagePullPolicy: IfNotPresent
          command: [ "sh", "-c", "sleep inf" ]
          resources:
            requests:
              cpu: 1000m
              memory: 2048Mi
          volumeMounts:
          - name: host-sys
            mountPath: /sys
          terminationMessagePath: /dev/termination-log
          securityContext:
            privileged: true
            seccompProfile:
              type: RuntimeDefault
            capabilities:
              add:
                - CAP_SYS_ADMIN
                - CAP_FOWNER
                - NET_ADMIN
                - SYS_ADMIN
              drop:
                - ALL
            runAsUser: 0
            runAsGroup: 0
            runAsNonRoot: false
            readOnlyRootFilesystem: false
            allowPrivilegeEscalation: true
      volumes:
      - name: host-sys
        hostPath:
          path: /sys
          type: Directory
      nodeName: master-0
      priorityClassName: system-cluster-critical
      hostPID: true
      hostIPC: true
      hostNetwork: true
    EOF
    
    1. Delete an existing Pod and apply a new Pod.
    oc delete pod/bpftrace-diagnostic-0; oc apply -f bpftrace-diagnostic-0.yaml
    
    1. Connect to the Pod
    oc rsh pod/bpftrace-diagnostic-0
    
    1. Check the file open with bpftrace
    dnf install -y bpftrace
    bpftrace -e 'tracepoint:syscalls:sys_enter_open  { printf("%s %s\n", comm, str(args->filename));}'
    

    You can see files open.

    ❯ bpftrace -e 'kprobe:vfs_open { printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name)); }' | grep so | grep cry
    open path: libgcrypt.so.20.2.5
    open path: libgcrypt.so.20.2.5
    open path: libgcrypt.so.20.2.5
    open path: libgcrypt.so.20.2.5
    open path: .libgcrypt.so.20.hmac
    open path: .libgcrypt.so.20.hmac
    open path: libcrypto.so.1.1.1k
    

    You can see FIPS checks.

    ❯ bpftrace -e 'kprobe:vfs_open { printf("open path: %s\n",                                  str(((struct path *)arg0)->dentry->d_name.name)); }' | grep fips         
    open path: fips_enabled
    open path: fips_enabled
    

    Link to bpftrace docs https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#2-kprobekretprobe-dynamic-tracing-kernel-level-arguments

    You can see the cool things happening on your Node.

  • Things I learned this week

    Things that I’ve learned for the week:

    1. containerbuildsystem/atomic-reactor recently released v4.5.0 which is a simple python library for building docker images – supporting OSBS 2.0 components.
    2. Redbooks: IBM Cloud Pak for Data on IBM zSystems is a high-level overview of IBM zSystems with Cloud Pak for Data.
    3. Redbooks: Introduction to IBM PowerVM introduces PowerVM virtualization technologies on Power servers.
    4. operator-framework/operator-sdk v1.28.0 is released. On a Mac, use brew upgrade operator-sdk.
    5. LinkedIn: Treating IBM Power like cattle will help you modernize! is a new blog from an IBMer which highlights a mentality switch:

    Changing how you view your solutions can free you to modernise, evolve, and detect the challenges that lie ahead from a far better vantage point.

    1. k8s.gcr.io being redirected and may have some consequences on your environment.

    On Monday, March 20th, traffic from the older k8s.gcr.io registry will be redirected to registry.k8s.io with the eventual goal of sunsetting k8s.gcr.io.

    1. I was told about a very cool link to see all the various SIGs link
  • Things for the Week

    This week I learned a few things of interest:

    In close collaboration with Red Hat the IBM Power Ecosystem team has continued efforts to enable and advance products running on the Power platform. Click here to review the new releases in February: https://community.ibm.com/community/user/powerdeveloper/blogs/ashwini-sule/2023/03/02/red-hat-products-february-2023-releases

    IBM Power Developer Exchange

    I found the list helpful when using the Power architecture and OpenShift.

    Want to develop applications with Red Hat OpenShift Dev Spaces but don’t know where to start? This blog outlines the step-by-step process for installing OpenShift Dev Spaces on the Red Hat OpenShift Container Platform on IBM Power: https://community.ibm.com/community/user/powerdeveloper/blogs/sachin-itagi/2023/03/03/developing-net-applications-on-ibm-power-using-vis 

    IBM Power Developer Exchange

    I haven’t stayed current on all the cool things in OpenShift, I thought this one held the most promise for end-to-end devops.

    I needed to figure out why my Worker’s networking was disconnected from the network:

    oc get nodes
    ssh core@osa21-worker-1.sslip.io
    nmcli device
    nmcli con reload env3
    nslookup quay.io

    After the restart the networking worked. It told me there was something wrong with the local networking, so I checked the DNS Operator. I had to restart the operator and make some changes to a DNS server that was actually up.

    If you hit some networking issues, the above will help.

    You can solve the multi-architecture multi-image problem when automating and sharing images across IBM Power and x86 with container manifests. Learn how here: https://community.ibm.com/community/user/powerdeveloper/viewdocument/build-multi-architecture-container?CommunityKey=2d4070a1-ca52-4e83-8efb-02b41c42459e&tab=librarydocuments 

    IBM Power Developer Exchange

    If you need to build manifest images the above is very helpful.

  • How to grab ignition files

    I was helping a colleague grab the latest ignition files for his OpenShift Container Platform workers.

    1. Connect to the bastion
    ❯ ssh root@<hostname>
    1. List the master nodes and select a master node
    ❯ oc get nodes -lnode-role.kubernetes.io/master=
    NAME                                                        STATUS   ROLES                  AGE     VERSION
    master-0.ocp-power.xyz   Ready    control-plane,master   5d19h   v1.25.2+7dab57f
    master-1.ocp-power.xyz   Ready    control-plane,master   5d19h   v1.25.2+7dab57f
    master-2.ocp-power.xyz   Ready    control-plane,master   5d19h   v1.25.2+7dab57f
    1. Get the IP address
    ❯ oc get nodes master-2.ocp-power.xyz -o json | jq -r .status.addresses
    [
      {
        "address": "192.168.100.240",
        "type": "InternalIP"
      },
      {
        "address": "master-2.ocp-power.xyz",
        "type": "Hostname"
      }
    ]
    1. The Machine Config Server has the latest igntion file
    ❯ curl https://192.168.100.240:22623/config/worker -k -H "Accept: application/vnd.coreos.ignition+json;version=3.2.0" | jq -r . > worker-$(date +%Y-%m-%d).ign

    worker can be replaced with master or any other MachineConfigPool

    The machine-config-server is hosted on each of the master nodes and the bootstrap node.

    Note, this makes it download ignition version 3.2.0.

    1. Download the ignition file
    ❯ scp root@<hostname>:'~'/worker-$(date +%Y-%m-%d).ign .
    worker-2023-03-02.ign                          100%  357KB 249.3KB/s   00:01    

    You can use this file for your work with worker ignition.

    Refer to https://github.com/openshift/machine-config-operator/blob/e39fea2d5c1d6991df4f7dd526b6292238f2ecfc/pkg/server/api.go#L196

  • Interesting Things of the Week for February 17, 2023

    Security Implementation with Red Hat OpenShift on IBM Power Systems

    …As with any production system, it is important to ensure the security of an OpenShift deployment. This includes secure deployment and configuration of the OpenShift components, as well as ongoing maintenance and monitoring to ensure the continued security of the environment. This Redpaper publication provides a comprehensive overview of the security best practices for deploying Red Hat OpenShift on IBM Power systems…

    IBM Redpaper: https://www.redbooks.ibm.com/redpieces/abstracts/redp5690.html

    The IBM RedPaper is pretty comprehensive on Securing OpenShift (note it’s a draft).

    Power Developer Exchange: Installing single node OpenShift to PowerVM

    CHONGSHI ZHANG shows how to install a SNO (single node OpenShift) to a PowerVM instance. It’s very powerful for development and other purposes.

    IBM Power Developer Exchange: https://community.ibm.com/community/user/powerdeveloper/blogs/chongshi-zhang/2023/02/09/installing-sno-to-powervm

    The Power Developer Exchange has a detailed approach to Single Node OpenShift.

    A script to rotate Encryption keys for etcd

    oc patch kubeapiserver cluster --type merge -p "
    spec:
      unsupportedConfigOverrides:
        encryption:
          reason: force KAS rotation `date`
    "
    Rotating Encryption Keys on OpenShift etcd https://patch-diff.githubusercontent.com/raw/ocp-power-automation/ocp4-playbooks-extras/pull/45.patch

    The above code is super handy for rotating the etcd encryption keys.

    Red Hat updated the oc-compliance kubectl plugin.

    RHEA-2023:0797 https://access.redhat.com/errata/RHEA-2023:0797

    oc-compliance is now updated.

    Error: creating build container: copying system image from manifest list: Source image rejected: None of the signatures were accepted

    [2/2] STEP 1/4: FROM registry.access.redhat.com/ubi8/ubi:8.7-1054.1675788412
    Trying to pull registry.access.redhat.com/ubi8/ubi:8.7-1054.1675788412...
    Error: creating build container: copying system image from manifest list: Source image rejected: None of the signatures were accepted, reasons: open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory; open /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta: no such file or directory

    You can then override the trust

    # podman image trust set -t reject default
    # podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release -t signedBy registry.access.redhat.com
    # podman image trust show 
    https://access.redhat.com/solutions/5525441

    The latest podman runs into some issues with trusting sources.

    Error: error copying image "78b2869b282bf2f28a5e873d6ade079e83d77765223c7bcd180b77cbc8fe4751": Source image rejected: Running image containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@78b2869b282bf2f28a5e873d6ade079e83d77765223c7bcd180b77cbc8fe4751 is rejected by policy.

    In podman, you might hit the above, and need to switch to insecureAcceptAnything when you do a podman push.

    You’ll want to modify /etc/containers/policy.json default from type reject to insecureAcceptAnything

        "default": [
            {
                "type": "insecureAcceptAnything"
            }
  • Interesting Things for the Week Ending 10 Feb 2023

    During the week, I accumulate a lot of links/information that I want to come back to over and over again. I find it helpful to blog about them. Here are some of things I found interesting this week, and frankly helpful.

    Build multi-arch images on GitHub Actions with Buildx 

    If you have ever wondered how to build multi-arch containers to run on ppc64le, x86, ARM, and/or s390x with GitHub Actions, then this article is for you. GitHub Actions is a continuous integration … 

    https://community.ibm.com/community/user/powerdeveloper/blogs/siddhesh-ghadi/2023/02/08/build-multi-arch-images-on-github-actions-with-bui

    A comprehensive article on using docker buildx to generate manifest listed builds.

    Routing Via Host with OVNKuberenetes 

    OVN-Kuberenetes is the default network provider for OpenShift Container Platform. OVN-Kubernetes is based on Open Virtual Network (OVN) and provides an overlay-based networking implementation. OVN-Kubernetes … 

    https://community.ibm.com/community/user/powerdeveloper/blogs/mick-tarsel/2023/01/26/routingviahost-with-ovnkuberenetes

    The background and why we use OVN-Kubernetes is superb. Mick did a bang up job on it.

    Making a OpenShift Container Platform StorageClass the default

    I had to recollect how to set the StorageClass as the default. You can check the current storageclass setting

    ❯ oc get storageclass
    NAME                   PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
    nfs-client             k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  18m

    Patch the storageclass name – nfs-client

    ❯ oc patch storageclass nfs-client -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

    Check the storage class is now the default

    ❯ oc get storageclass
    NAME                   PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
    nfs-client (default)   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  18m
    

    References

    1. Changing the default storage class

    The above is a fast way to set the StorageClass default.

    Setting Namespace Security Labels

    In case you need to set it for a specific namespace such as nfs-provisioner.

    oc label namespace/nfs-provisioner security.openshift.io/scc.podSecurityLabelSync=false --overwrite=true
    oc label namespace/nfs-provisioner pod-security.kubernetes.io/enforce=privileged --overwrite=true
    oc label namespace/nfs-provisioner pod-security.kubernetes.io/audit=privileged --overwrite=true
    oc label namespace/nfs-provisioner pod-security.kubernetes.io/warn=privileged --overwrite=true

    The above is a quick macro for setting it for a given namespace so you can get privileged access.

  • Notes on qcow2 on centos

    I recently had to run a centos9 qcow2 on a centos7 machine. I ran into a few problems, however, I found these steps helpful as I worked through the issue and resolved my problem. I’ve recorded them here for posterity.

    Steps

    1. Navigate to https://cloud.centos.org/centos/9-stream/x86_64/images/
    2. Click Last Modified twice to sort the images from most recent to oldest
    3. Find the latest qcow2 image – CentOS-Stream-GenericCloud-9-20230207.0.x86_64.qcow2
    4. Right Click and Copy Link
    https://cloud.centos.org/centos/9-stream/x86_64/images/
    1. Connect to your host
    ❯ curl -O -L https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20230207.0.x86_64.qcow2
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  930M  100  930M    0     0  63.2M      0  0:00:14  0:00:14 --:--:--  104M
    1. Install the dependencies
    ❯ dnf install libguestfs-tools qemu-kvm.x86_64 libvirt virt-install libguestfs-xfs.x86_64
    CentOS-7 - Base         0.0  B/s |   0  B     00:00    
    CentOS-7 - Updates      0.0  B/s |   0  B     00:00    
    CentOS-7 - Extras       0.0  B/s |   0  B     00:00    
    Package libguestfs-tools-1:1.40.2-10.el7.noarch is already installed.
    Package qemu-kvm-10:1.5.3-175.el7_9.6.x86_64 is already installed.
    Package libvirt-4.5.0-36.el7_9.5.x86_64 is already installed.
    Package virt-install-1.5.0-7.el7.noarch is already installed.
    Dependencies resolved.
    Nothing to do.
    Complete!
    1. Move the qcow over to images
    ❯ mv CentOS-Stream-GenericCloud-9-20230207.0.x86_64.qcow2 /var/lib/libvirt/images/
    1. Generate a password
    ❯ openssl rand -hex 10
    037c94bb31a9b9870178-example
    1. Set the password based on the previous step’s output
    ❯ LIBGUESTFS_BACKEND=direct virt-customize --format qcow2 -a /var/lib/libvirt/images/CentOS-Stream-GenericCloud-9-20230207.0.x86_64.qcow2 --root-password password:037c94bb31a9b9870178-example

    Note, if it fails, add -v -x to see verbose logging. Also make sure your base OS is one that can process the filesystem and run the qcow2 image. E.g. RHEL8 or Centos8 or higher.

    1. Startup the VM
    ❯ sudo virt-install
        --name ocp-bastion-server
        --ram 4096
        --vcpus 2
        --disk path=/var/lib/libvirt/images/CentOS-Stream-GenericCloud-9-20230207.0.x86_64.qcow2 
        --os-type linux
        --os-variant rhel9.0
        --network bridge=virbr0
        --graphics none
        --serial pty
        --console pty
        --boot hd
        --import

    References

    1. https://kubevirt.io/2020/Customizing-images-for-containerized-vms.html#building-standard-centos-8-image
    2. https://forums.centos.org/viewtopic.php?t=78770
    3. https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/
    4. https://www.reddit.com/r/CentOS/comments/k5sz8h/centos_8_image_editing_withing_centos7_host/

    To Grab RHCOS 4.12.

    1. Download from the mirror
    ❯ curl -O -L https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.12/4.12.2/rhcos-qemu.x86_64.qcow2.gz
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 1149M  100 1149M    0     0  25.5M      0  0:00:45  0:00:45 --:--:-- 32.1M
    1. Unzip
    ❯ gunzip rhcos-qemu.x86_64.qcow2.gz

    Debugging the FileSystem

    If you have the wrong version installed, sometimes the file system echos issues with superblock.

    guestfish -a /var/lib/libvirt/images/CentOS-Stream-GenericCloud-9-20230207.0.x86_64.qcow2 
    run 
    list-filesystems
    mount /dev/sda1 /
    dmesg | tail
    ><fs> run
    ><fs> list-filesystems
    /dev/sda1: xfs
    ><fs> mount /dev/sda1 /
    libguestfs: error: mount: mount exited with status 32: mount: wrong fs type, bad option, bad superblock on /dev/sda1, missing codepage or helper program, or other error In some cases useful info is found in syslog - try
           dmesg | tail or so.
    ><fs> dmesg | tail
    [   19.169691]  sda: sda1
    [   19.191795]  sda: sda1
    [   19.211130]  sda: sda1
    [   19.232340]  sda: sda1
    [   76.488398] SGI XFS with ACLs, security attributes, no debug enabled
    [   76.493455] XFS (sda1): Superblock has unknown read-only compatible features (0x4) enabled.
    [   76.504604] XFS (sda1): Attempted to mount read-only compatible filesystem read-write.
    [   76.505325] XFS (sda1): Filesystem can only be safely mounted read only.
    [   76.505362] XFS (sda1): SB validate failed with error -22.
    ><fs> 
    ><fs> quit
  • Cool Things I learned last week

    For those following along with my work, I’ve compiled a list of interesting items I’ve run across in the last week:

    Install minikube on an IBM PowerVM running RHEL 8.6 or 8.7

    Want to learn how to install minikube on an IBM Power system running RHEL? Check out this new blog on the IBM Power Developer eXchange, which provides step-by-step instructions on how to identify the software dependencies needed to download, build, and install minikube on Power

    https://community.ibm.com/community/user/powerdeveloper/blogs/vijay-puliyala/2023/01/23/install-minikube-on-ibm-powervm

    Learn the Compliance Operator

    There is a nice self-paced lab to learn the compliance-operator

    https://github.com/JAORMX/lab-compliance-operator
  • Compliance Operator Quick Notes

    If you’re processing Credit Card Payments on the OpenShift Container Platform, the Payment Card Industry and the Data Security Standard is a must on your cluster.

    With Red Hat’s release of the Compliance Operator v0.1.59, they added support for IBM Power Systems.

    I wanted to share a couple of notes about the Compliance Operator:

    1. The Compliance Operator supports the ocp4-cisocp4-cis-nodeocp4-pci-dss-node and ocp4-pci-dss profiles.
    2. The Compliance Operator sources the profiles which have a set of rules from ComplianceAsCode/content. For instance, you can see the OpenShift profiles at products/ocp4/profiles/.
    3. The Compliance Operator PCI-DSS profiles support v3.2.1.
    4. If you see a profile with a postfix of -node, it’s focus is on the Operating System.
    5. If you see no -node on the profile name, it’s focus is on the Kubernetes and OpenShift platform.

    Clarification 2023-FEB-07 I learned the -node and node were actually there due to limitations in oscap, together they represent the same profile and are expected to be applied together.

    I hope this quick notes help you.

    References
    1. Medium/Aditi Jadhav: Using the Compliance Operator to support PCI-DSS on OpenShift Container Platform on Power
    2. Power Developer Exchange: OpenShift Compliance Operator 0.1.59 now supports PCI-DSS on Power
    3. PCI-DSS v3.2.1 Standard
    4. Supported compliance profiles
  • My MachineConfigPool is … Stuck

    My teammate was investigating an SSHD config change and hit a stuck MachineConfigPool. Here are some steps we followed to get it unstuck.

    Steps

    1. Verify that the MachineConfigPool is stuck updating
    ❯ oc get mcp
    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master   rendered-master-0de63bfa1c0db0777031adddb3286fbc   False     True       True       3              0                   0                     3                      9d
    worker   rendered-worker-38e4049eaf0b7fca848408378092e607   True      False      False      3              3                   3                     0                      9d
    
    1. Find out for one of your nodes in the mcp that is stuck (for instance, master-0)
    ❯ oc get pods -n openshift-machine-config-operator --field-selector spec.nodeName=master-0
    NAME                          READY   STATUS    RESTARTS   AGE
    machine-config-daemon-t8x8j   2/2     Running   2          35h
    machine-config-server-kfx8n   1/1     Running   1          35h
    
    1. Check the logs and grab the rendered-master
    ❯ oc logs pod/machine-config-daemon-tgnss -n openshift-machine-config-operator
    ...
    E0124 07:19:26.746977  780508 on_disk_validation.go:208] content mismatch for file "/etc/ssh/sshd_config" (-want +got):
      bytes.Join({
    -       "\n#\t",
    +       "#       ",
            "$OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $\n\n# Th",
            "is is the sshd server system-wide configuration file.  See\n# ssh",
            ... // 1437 identical bytes
            "keys and .ssh/authorized_keys2\n# but this is overridden so insta",
            "llations will only check .ssh/authorized_keys\nAuthorizedKeysFile",
    -       `       `,
    +       "      ",
            ".ssh/authorized_keys\n\n#AuthorizedPrincipalsFile none\n\n#Authorize",
            "dKeysCommand none\n#AuthorizedKeysCommandUser nobody\n\n# For this ",
            ... // 2258 identical bytes
            "E LC_MEASUREMENT\nAcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE\nAcc",
    ...
    +       "\n",
      }, "")
    E0124 07:19:26.747042  780508 writer.go:200] Marking Degraded due to: unexpected on-disk state validating against rendered-master-0de63bfa1c0db0777031adddb3286fbc: content mismatch for file "/etc/ssh/sshd_config"
    I0124 07:19:28.973484  780508 daemon.go:1248] Current+desired config: rendered-master-0de63bfa1c0db0777031adddb3286fbc
    ...
    
    1. OK, this looks like a problem with the whitespace, and inspect the URL decoded version’s whites pace vim :set list
    > oc get mc rendered-master-0de63bfa1c0db0777031adddb3286fbc -o yaml > out.yaml
    

    You may have to update the white space.

    1. Check the reasons for the failure if the whitespace doesn’t fix it.
    > oc describe mcp master
    
    Message:
        Node master-0 is reporting: 
            "unexpected on-disk state validating against rendered-master-0de63bfa1c0db0777031adddb3286fbc: 
            mode mismatch for file: \"/etc/ssh/sshd_config\"; 
            expected: -rw-------/384/0600; received: -rw-r--r--/420/0644", 
            Node master-1 is reporting: "unexpected on-disk state validating 
            against rendered-master-0de63bfa1c0db0777031adddb3286fbc: content 
            mismatch for file \"/etc/ssh/sshd_config\"", Node master-2 is reporting:
            "unexpected on-disk state validating against 
            rendered-master-0de63bfa1c0db0777031adddb3286fbc: content mismatch for file 
            \"/etc/ssh/sshd_config\""
    

    In this case, the local files were edited while preparing the ideal sshd_config and needed a forced update.

    1. Force the machine-config to refresh files.
    > touch /run/machine-config-daemon-force
    1. You should see the states change after the node reboots.
    Events:
      Type    Reason            Age    From                                    Message
      ----    ------            ----   ----                                    -------
      Normal  AnnotationChange  5m19s  machineconfigcontroller-nodecontroller  Node master-0 now has machineconfiguration.openshift.io/state=Done
    
      degradedMachineCount: 2
      machineCount: 3
      observedGeneration: 500
      readyMachineCount: 0
      unavailableMachineCount: 2
      updatedMachineCount: 0

    If you need to select a file from the rendered config:

    > oc get mc rendered-master-0de63bfa1c0db0777031adddb3286fbc -o yaml | yq -r '.spec.config[].files[] | select(.path == "/etc/ssh/sshd_config").contents.source'
    data:,%0A%23%09$OpenBSD:%20sshd_config%2Cv%201.103
    ...

    References