Blog

There are two ways to grab the ignition files for the workers in the cluster:

1. A downloaded ignition file stored (in data folder) using curl:

• curl -k http://api.demo.ocp-multiarch.xyz:22623/config/worker -o worker.ign -H "Accept: application/vnd.coreos.ignition+json;version=3.2.0"

2. Download the ignition file using the oc commandline

• oc extract -n openshift-machine-api secret/worker-user-data --keys=userData --to=-

I’m adding this because I use it every day, and others might find it helpful.

Here are some lessons learned from the week (and shared with other):

TIP: How to create a Power Workspace from the CLI

This document outlines the steps necessary to create a PowerVS workspace.

1. Login to IBM Cloud

ibmcloud login --sso -r jp-osa -c 65b64c111111114bbfbd893c2c

• -r jp-osa targets the region
• -c 65b64c111111114bbfbd893c2c is the account that is being targetted.

2. Install the Plugins for PowerVS and VPC.

As a good practice install, the powervs and vpc plugins.

❯ ibmcloud plugin install power-iaas -f
❯ ibmcloud plugin install 'vpc-infrastructure' -f 

3. Create a Power Systems workspace

1. The two elements you should configure below are the PVS_REGION and the WORKSPACE_NAME. The rest will create a new Workspace.

WORKSPACE_NAME=rdr-mac-osa-n1
SERVICE_NAME=power-iaas
RESOURCE_GROUP_NAME=my-resource-group
SERVICE_PLAN_NAME=power-virtual-server-group
PVS_REGION=osa21
❯ ibmcloud resource service-instance-create \
    "${WORKSPACE_NAME}" \
    "${SERVICE_NAME}" \
    "${SERVICE_PLAN_NAME}" \
    "${PVS_REGION}" \
    -g "${RESOURCE_GROUP_NAME}"
Creating service instance rdr-mac-osa-n1 in resource group my-resource-group of account Power Account as pb@ibm.xyz...
OK
Service instance rdr-mac-osa-n1 was created.
                  
Name:             rdr-mac-osa-n1
ID:               crn:v1:bluemix:public:power-iaas:osa21:a/65b64c1f1c29460e8c2e4bbfbd893c2c:e3772ff7-48eb-4c81-9ee0-07cf0b5547a5::
GUID:             e3772ff7-48eb-4c81-9ee0-07cf0b5547a5
Location:         osa21
State:            provisioning
Type:             service_instance
Sub Type:         
Allow Cleanup:    false
Locked:           false
Created at:       2023-07-11T14:12:07Z
Updated at:       2023-07-11T14:12:10Z
Last Operation:             
                  Status    create in progress
                  Message   Started create instance operation

Thus you are able to create a PowerVS Workspace from the CLI.

Note: Flatcar Linux

Flatcar Container Linux A community Linux distribution designed for container workloads, with high security and low maintenance

https://www.flatcar.org/

I learned about flatcar linux this week… exciting stuff.

Demonstration: Tang Server Automation on PowerVM

I work on the OpenShift Container Platform on Power Team which created a few video to discuss The powervm-tang-server-automation project provides Terraform based automation code to help with the deployment of Network Bound Disk Encryption (NBDE) on IBM® Power Systems™ virtualization and cloud management. Thanks to Aditi Jadhav for presenting.

The Overview

The Demonstration

https://community.ibm.com/community/user/powerdeveloper/blogs/paul-bastide/2023/07/12/tang-server-automation-on-powervm

New Blog: Installing OpenShift on IBM Power Virtual Servers with IPI

My colleague Ashwin posted a new blog using IPI – This blog covers creating and destroying a “public” OpenShift cluster i.e., one which is accessible through the internet and the nodes of the cluster can access the internet. 

https://community.ibm.com/community/user/powerdeveloper/blogs/ashwin-hendre/2023/07/10/powervs-ipi

Here are some notes from the week:

The Network Observability Operator is now released on IBM Power Systems.

Network observability operator now available on Power.

Network Observability (NETOBSERV) for IBM Power, little endian 1 for RHEL 9 ppc64le

https://access.redhat.com/errata/RHSA-2023:3905?sc_cid=701600000006NHXAA2 https://community.ibm.com/community/user/powerdeveloper/discussion/network-observability-operator-now-available-on-power

Here is a compendium of tidbits from the week:

FYI: Terraform v1.5.0 is released. It’s probably worth updating. link to ppc64le build

❯ brew upgrade terraform

There are new features:

check blocks for validating infrastructure: Module and configuration authors can now write independent check blocks within their configuration to validate assertions about their infrastructure. Adds a new strcontains function that checks whether a given string contains a given substring. (#33069)

https://github.com/hashicorp/terraform/releases/tag/v1.5.0

Blog: Build multi-architecture container images on GitHub Actions using native nodes

My colleague, Yussuf, has posted a blog on using GitHub actions with Native Nodes. It’s to the point, and super helpful. link

There are already a few good blogs[1][2][3] available in this group that demonstrates how to build multi-arch container images on GitHub Actions using Buildx. However, they are using QEMU which is a free and open-source emulator for running cross-builds. Using QEMU presents its own problems where the main point is the slowness which cannot match when we run the builds natively. Even there is no guarantee that the build will always succeed when we use low-level code in the project. These pain points forced us to use native nodes as part of the same Buildx workflow inside a GitHub Action. If you as well want to use native nodes to build your projects on multiple architectures including ppc64le then this article is for you.

https://community.ibm.com/community/user/powerdeveloper/blogs/yussuf-shaikh/2023/06/14/workflow-using-buildx-remote-builders

Tip: Custom Build of QEMU on Centos 9

I had to do a cross-build of ARM64, so I used QEMU. It’s not necessarily straight-forward on CENTOS9, here are the steps I took:

1. Connect to my machine.

❯ ssh cloud-user@<machine IP>

2. Switch to Root

❯ sudo -s 

3. Enable the Code Ready Builders repo

❯ dnf config-manager --set-enabled crb

4. install a bunch of dependencies

❯ dnf install -y git make pip vim ninja-build gcc glib2-devel.x86_64 pixman.x86_64 libjpeg-devel giflib-devel pixman-devel cairo-devel pango-devel qemu-kvm edk2-aarch64

5. Install Python dependencies

❯ pip install Sphinx sphinx-rtd-theme

6. Pull the Qemu Code

❯ git clone https://git.qemu.org/git/qemu.git

7. Change to QEMU

❯ cd qemu

8. Apply the patch

From 14920d35f053c8effd17a232b5144ef43465a85e Mon Sep 17 00:00:00 2001
From: root <root@cross-build-pbastide.novalocal>
Date: Tue, 20 Jun 2023 10:02:39 -0400
Subject: [PATCH] test

---
 configure | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index 01a53576a7..02cabb556b 100755
--- a/configure
+++ b/configure
@@ -371,6 +371,7 @@ else
   # be the result of a missing compiler.
   targetos=bogus
 fi
+targetos=linux
 
 # OS specific
 
@@ -508,7 +509,7 @@ case "$cpu" in
   sparc64)
     CPU_CFLAGS="-m64 -mcpu=ultrasparc" ;;
 esac
-
+CPU_CFLAGS="-m64 -mcx16"
 check_py_version() {
     # We require python >= 3.7.
     # NB: a True python conditional creates a non-zero return code (Failure)
-- 

The above patch is generated from git format-patch -1 HEAD.

9. Create the build directory

❯ mkdir -p build

9. Configure the qemu build

❯ ./configure --target-list=aarch64-softmmu --enable-virtfs --enable-slirp

10. Build with all your processors.

❯ make -j`nproc`

11. The file should exist and be in your native architecture.

❯ file ./build/qemu-system-aarch64
./build/qemu-system-aarch64: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1871423864de4428c4721b8e805304f0142bed6a, for GNU/Linux 3.2.0, with debug_info, not stripped

12. Download Centos qcow2 from link

❯ curl -O https://cloud.centos.org/centos/9-stream/aarch64/images/CentOS-Stream-GenericCloud-9-20220621.1.aarch64.qcow2

13. Update the .ssh/authorized_keys

modprobe nbd
qemu-nbd -c /dev/nbd0 CentOS-Stream-GenericCloud-9-20220621.1.aarch64.qcow2
mount /dev/nbd0p1 /mnt
mkdir -p /mnt/root/.ssh/

14. Edit the keys to add your public key.

vim /mnt/root/.ssh/authorized_keys

15. Unmount the qcow2 mount

umount /mnt
qemu-nbd -d /dev/nbd0

16. Start the QEMU vm.

build/qemu-system-aarch64 -m 4G -M virt -cpu cortex-a57 \
  -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
  -drive if=none,file=CentOS-Stream-GenericCloud-9-20220621.1.aarch64.qcow2,id=hd0 \
  -device virtio-blk-device,drive=hd0 \
  -device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp:127.0.0.1:5555-:22 \
  -nographic

Finally, you’ll see:

CentOS Stream 9
Kernel 5.14.0-115.el9.aarch64 on an aarch64

Activate the web console with: systemctl enable --now cockpit.socket

cross-build-pbastide login: 

It’s ready-to-go.

References

1. http://cdn.kernel.org/pub/linux/kernel/people/will/docs/qemu/qemu-arm64-howto.html
2. https://fedoraproject.org/wiki/Architectures/AArch64/Install_with_QEMU
3. https://wiki-archive.linaro.org/LEG/UEFIforQEMU
4. https://packages.debian.org/experimental/qemu-efi-aarch64
5. https://www.redhat.com/sysadmin/install-epel-linux
6. https://wiki.qemu.org/Hosts/Linux
7. https://github.com/Automattic/node-canvas/issues/1065#issuecomment-1278496824
8. https://wiki.debian.org/Arm64Qemu

I’ve been working hard on multiarch enablement for various OpenShift features. Here are a few notes from the last few weeks:

New Blog on Cluster API

Delve into the powerful capabilities of Cluster API and how it enables effortless K8s cluster deployment on PowerVC: https://community.ibm.com/community/user/powerdeveloper/blogs/prajyot-parab/2023/05/31/simplifying-k8s-cluster-deployment-leveraging-capi 

Prajyot Parat from the Kubernetes/OpenShift on Power team

It’s a really helpful and interesting solution to deploying your cluster.

TIP: OpenShift Installer Provisioned Infrastructure on IBM Cloud

I installed a new cluster with the openshift-installer using IPI on IBM Cloud with a pre-defined VPC with predefined networks. If your install hangs and fails mysteriously after 30-40 minutes with three provisioned RHCOS nodes trying to call out to quay.io, it could point to the Public Gateway for the network not being enabled so it can call back to quay.io.

This issue was tough to debug, and I hope it helps you.

TIP: scp hangs because of bad mtu

The scp command opens the channel and hangs…

scp -vvv -i data/id_rsa sample.txt root@1.1.1.1:/tmp
...
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 6/7/8 sock -1 cc -1)

You can go check the optimal MTU to send to the destination.

# ping 1.1.1.1 -c 10 -M do -s 1499
PING 1.1.1.1 (1.1.1.1) 1499(1527) bytes of data.
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500

Then per the link https://unix.stackexchange.com/questions/14187/why-does-scp-hang-on-copying-files-larger-than-1405-bytes

ip link set eth0 mtu 1400

Then it’ll work.

The above will help when scp hangs.

Blog: Using the oc-compliance plugin on the OpenShift Container Platform on Power

My team has added support for oc-compliance on OpenShift Container Platform on IBM Power, and in this post, I’m sharing the download, the setup, and using the tool in the cluster. 

https://community.ibm.com/community/user/powerdeveloper/blogs/aditi-jadhav/2023/06/07/using-the-oc-compliance-plugin-on-the-openshift-co

The oc-compliance plugin is super helpful, and my colleague Aditi has created a new blog on oc-compliance.

Blog: Configuring Seccomp Profile on OpenShift Container Platform for Security and Compliance on Power

A blog on using seccomp with OCP4.

https://medium.com/@aditijadhav38/configuring-seccomp-profile-on-openshift-container-platform-for-security-and-compliance-on-power-d94907f4b1f9

My teammate Aditi updated for 4.12 and 4.13 (surprise no changes, which is good).