Multi-Arch Compute for Red Hat OpenShift Container Platform on IBM Power systems lets one use a pair of compute architectures, such as, ppc64le and amd64, within a single cluster. This feature opens new possibilities for versatility and optimization for composite solutions that span multiple architectures. The cluster owner is able to add an additional worker post installation.
With User Provisioned Infrastructure (UPI), the cluster owner may have used automation or manual setup of front-end load balancers. The IBM team provides PowerVS ocp4-upi-powervs, PowerVM ocp4-upi-powervm and HMC ocp4-upi-powervm-hmc automation.
When installing a cluster, the cluster is setup with ab external load balancer, such as haproxy. The external load balancer routes traffic to pools the Ingress Pods, API Server and MachineConfig server. The haproxy configuration is stored at /etc/haproxy/haproxy.cfg.
For instance, the configuration for ingress-https load balancer would look like the following:
frontend ingress-https
bind *:443
default_backend ingress-https
mode tcp
option tcplog
backend ingress-https
balance source
mode tcp
server master0 10.17.15.11:443 check
server master1 10.17.19.70:443 check
server master2 10.17.22.204:443 check
server worker0 10.17.26.89:443 check
server worker1 10.17.30.71:443 check
server worker2 10.17.30.225:443 check
When adding a post-installation worker to a UPI cluster, one must update the ingress-http and ingress-https. Y
a. Find backend ingress-http then before the first server entry add the worker hostnames and ips.
server worker-amd64-0 10.17.15.11:80 check
server worker-amd64-1 10.17.19.70:80 check
b. Find backend ingress-https then before the first server entry add the worker hostnames and ips.
server worker-amd64-0 10.17.15.11:443 check
server worker-amd64-1 10.17.19.70:443 check
c. Save the config file.
Restart the haproxy
# systemctl restart haproxy
You now have the additional workers incorporated into the haproxy, and as the ingress pods are moved from Power to Intel and back. You have a fully functional environment.
P.P.S If you are running very advanced scenarios, you can change the ingresscontroller spec.nodePlacement.nodeSelector to put the workload on specific architectures. see Configuring an Ingress Controller
Red Hat OpenShift 4.14, Multi-Architecture Compute was introduced for the IBM Power and IBM Z platforms, enabling a single heterogeneous cluster across different compute architectures. With the release of Red Hat OpenShift 4.15, clients can now add x86 compute nodes to a multi-architecture enabled cluster running on Power. This simplifies deployment across different environments even further and provides a more consistent management experience. Clients are accelerating their modernization journeys with multi-architecture compute and Red Hat OpenShift by exploiting the best-fit architecture for different solutions and reducing cost and complexity of workloads that require multiple compute architectures.
The Red Hat OpenShift Container Platform runs on IBM Power systems, offering a secure and reliable foundation for modernizing applications and running containerized workloads. Multi-Arch Compute for OpenShift Container Platform lets you use a pair of compute architectures such as, ppc64le and amd64, within a single cluster. This exciting feature opens new possibilities for versatility and optimization for composite solutions that span multiple architectures. Join Paul Bastide, IBM Senior Software Engineer, as he introduces the background behind Multi-Arch Compute and then gets you started setting up, configuring, and scheduling workloads. After, Paul will take you through a brief demonstration showing common problems and solutions for running multiple architectures in the same cluster. This presentation sets the background and gets you started so you can set up, configure, and scheduling workloads. There will be a brief demonstration showing common problems and solutions for running multiple architectures in the same cluster.
Please join me on 11 April 2024, 9:00 AM ET. Please share any questions by clicking on the Reply button. If you have not done so already, register here and download it to your calendar.
Kube-burner is a Kubernetes performance and scale test orchestration toolset. It provides multi-faceted functionality, the most important of which are summarized below. A new version v1.9.2 is released.
FYI: How to visualize your OpenSCAP compliance reports Discover SCAPinoculars, a tool that helps you to visualize OpenSCAP reports, and the advantages it brings when used with the OpenShift Compliance Operator.
The Power Developer Exchange article dives into using the Red Hat Ansible Automation Platform and how to create PowerVS instances with Ansible. The collection is available at https://github.com/IBM-Cloud/ansible-collection-ibm
Per the blog, you learn to start a sample controller UI and running some sample program such as hello_world.yaml playbook to say hello to Ansible. With Ansible the options are infinite, and there is always something more to explore. We would like to know how you are using this solution, so drop us a comment.
The cluster wasn’t getting loaded, so I checked the following…. and it pointed to an issue of a call back to a cluster inside my firewall setup. The klusterlet shows that it’s an issue with a callback.
oc get pod -n open-cluster-management-agent
❯ oc get klusterlet klusterlet -oyaml Failed to create &SelfSubjectAccessReview{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:SelfSubjectAccessReviewSpec{ResourceAttributes:&ResourceAttributes{Namespace:,Verb:create,Group:cluster.open-cluster-management.io,Version:,Resource:managedclusters,Subresource:,Name:,},NonResourceAttributes:nil,},Status:SubjectAccessReviewStatus{Allowed:false,Reason:,EvaluationError:,Denied:false,},} with bootstrap secret “open-cluster-management-agent” “bootstrap-hub-kubeconfig”: Post “https://api.<XYZ>.com:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews”: dial tcp: lookup api.acmfunc.cp.fyre.ibm.com on 172.30.0.10:53: no such host
Install ibmcloud cli curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
Install the Power IAAS, Transit Gateway, Cloud Internet Services, and Infrastructure Service plugins ibmcloud plugin install power-iaas tg-cli vpc-infrastructure cis
Login to ibmcloud cli ibmcloud login --apikey API_KEY -r us-east
List the datacenters ibmcloud pi datacenters in our case we want wdc06
List the resource group id ❯ ibmcloud resource group dev-resource-group
❯ ibmcloud resource group dev-resource-group
Retrieving resource group dev-resource-group under account 555555555555555 as email@id.xyz...
OK
Name: dev-resource-group
Account ID: 555555555555555
ID: 44444444444444444
Default Resource Group: false
State: ACTIVE
Create a Workspace on a Power Edge Router enabled PowerVS zone. ibmcloud pi workspace-create rdr-mac-p2-wdc06 --datacenter wdc06 --group 44444444444444444 --plan public
❯ ibmcloud pi workspace-create rdr-mac-p2-wdc06 --datacenter wdc06 --group 44444444444444444 --plan public
Creating workspace rdr-mac-p2-wdc06...
Name rdr-mac-p2-wdc06
Plan ID f165dd34-3a40-423b-9d95-e90a23f724dd
❯ ibmcloud pi service-target crn:v1:bluemix:public:power-iaas:wdc06:a/555555555555555:7777777-6666-5555-44444-1111111::
Targeting service crn:v1:bluemix:public:power-iaas:wdc06:a/555555555555555:7777777-6666-5555-44444-1111111::...
Create a Power Network using the CRN so there is an IP Range for the Power workers.
❯ ibmcloud pi network-create-private ocp-net --dns-servers 9.9.9.9 --jumbo --cidr-block 192.168.200.0/24 --gateway 192.168.200.1 --ip-range 192.168.200.10-192.168.200.250
Creating network ocp-net under account Power Cloud - pcloudci as user email@id.xyz...
Network ocp-net created.
ID 3e1add7e-1a12-4a50-9325-87f957b0cd63
Name ocp-net
Type vlan
VLAN 797
CIDR Block 192.168.200.0/24
IP Range [192.168.200.10 192.168.200.250]
Gateway 192.168.200.1
DNS 9.9.9.9, 161.26.0.10, 161.26.0.11
Import the Centos8 stock image
❯ ibmcloud pi image-create CentOS-Stream-8
Creating new image from CentOS-Stream-8 under account Power Cloud - pcloudci as user email@id.xyz...
Image created from CentOS-Stream-8.
Image 4904b3db-1dde-4f3c-a696-92f068816f6f
Name CentOS-Stream-8
Arch ppc64
Container Format bare
Disk Format raw
Hypervisor phyp
Type stock
OS rhel
Size 120
Created 2024-01-24T21:00:29.000Z
Last Updated 2024-01-24T21:00:29.000Z
Description
Storage Type
Storage Pool
Find the closest location.
❯ ibmcloud tg locations
Listing Transit Service locations under account Power Cloud - pcloudci as user email@id.xyz...
OK
Location Location Type Billing Location
eu-es region eu
eu-de region eu
au-syd region ap
eu-gb region eu
br-sao region br
jp-osa region ap
jp-tok region ap
ca-tor region ca
us-south region us
us-east region us
❯ ibmcloud is vpc rdr-mac-p2-wdc06-vpc --output json | jq -r '.status'
available
Add a subnet
❯ ibmcloud is subnet-create sn01 rdr-mac-p2-wdc06-vpc \
--resource-group-id 44444444444444444 \
--ipv4-address-count 256 --zone us-east-1
Creating subnet sn01 in resource group 44444444444444444 under account Power Cloud - pcloudci as user email@id.xyz...
ID 0757-46e9ca2e-4c63-4bce-8793-f04251d9bdb3
Name sn01
CRN crn:v1:bluemix:public:is:us-east-1:a/555555555555555::subnet:0757-46e9ca2e-4c63-4bce-8793-f04251d9bdb3
Status pending
IPv4 CIDR 10.241.0.0/24
Address available 251
Address total 256
Zone us-east-1
Created 2024-01-24T16:18:10-05:00
ACL ID Name
r001-0a0afc6c-0943-4a0f-b998-e5e87ec93668 causation-browse-capture-behind
Routing table ID Name
r001-216fb1f5-da8f-447e-8515-649bc76b83aa retaining-acquaint-retiring-curry
Public Gateway -
VPC ID Name
r001-372372bb-5f18-4e36-8b39-4444444333 rdr-mac-p2-wdc06-vpc
Resource group ID Name
44444444444444444 dev-resource-group
❯ ibmcloud is subnet-update sn01 --vpc rdr-mac-p2-wdc06-vpc \
--pgw gw01
Updating subnet sn01 under account Power Cloud - pcloudci as user email@id.xyz...
ID 0757-46e9ca2e-4c63-4bce-8793-f04251d9bdb3
Name sn01
CRN crn:v1:bluemix:public:is:us-east-1:a/555555555555555::subnet:0757-46e9ca2e-4c63-4bce-8793-f04251d9bdb3
Status pending
IPv4 CIDR 10.241.0.0/24
Address available 251
Address total 256
Zone us-east-1
Created 2024-01-24T16:18:10-05:00
ACL ID Name
r001-0a0afc6c-0943-4a0f-b998-e5e87ec93668 causation-browse-capture-behind
Routing table ID Name
r001-216fb1f5-da8f-447e-8515-649bc76b83aa retaining-acquaint-retiring-curry
Public Gateway ID Name
r001-f5f27e42-aed6-4b1a-b121-f234e5149416 gw01
VPC ID Name
r001-372372bb-5f18-4e36-8b39-4444444333 rdr-mac-p2-wdc06-vpc
Resource group ID Name
44444444444444444 dev-resource-group
The IBM Power development team is happy to introduce cert-manager Operator for Red Hat OpenShift on Power. cert-manager is a “cluster-wide service that provides application certificate lifecycle management”. This service manages certfificates and integration with external certificate authorities using Automated Certificate Management Environment (ACME).
For v1.13, the release notes also tell you about the expanded support includes multiple architectures – AMD64, IBM Z® (s390x), IBM Power® (ppc64le) and ARM64 architectures.
This is exciting and I’ll give you a flavor of how to use the cert-manager with your OpenShift cluster. I’ll demonstrate how to use Let’s Encrypt for the HTTP01 challenge type and IBM Cloud Internet Services paired with Let’s Encrypt for the DNS01 challenge type.
This write up uses a 4.13 cluster on IBM PowerVS using ocp4-upi-powervs, the same steps apply to 4.14 and on-premises environments. To facilitate the HTTP01 challenge type, the IBM Cloud Services section is used:
### Using IBM Cloud Services
use_ibm_cloud_services = true
ibm_cloud_vpc_name = "rdr-cert-manager-vpc"
ibm_cloud_vpc_subnet_name = "sn-20231206-01"
ibm_cloud_resource_group = "resource-group"
iaas_vpc_region = "au-syd" # if empty, will default to ibmcloud_region.
ibm_cloud_cis_crn = "crn:v1:bluemix:public:internet-svcs:global:a/<account_id>:<cis_instance_id>::"
ibm_cloud_tgw = "rdr-sec-certman" # Name of existing Transit Gateway where VPC and PowerVS targets are already added.
This means you would have a CIS instance setup with a real domain linked. You would configure the IBM Cloud VPC to connect to the PowerVS workspace over a Transit Gateway. Ideally the connection uses the PER networking feature of PowerVS. This sets up a real hostname for the call back from Lets Encrypt and configures the Load Balancers which support port 80/443 traffic.
To setup the cert-manager, login to the Web Console as an administrator.
Click on Operators > OperatorHub
Filter on cert-manager
Select cert-manager for Red Hat OpenShift
Click Install using the namespace provided
Wait a few minutes for it to install.
You now have a working cert-manager operator, and ready for the HTTP01 challenge type. For this, we switch to the commandline.
Note, the above is a production letsencrypt, you could use staging. Be carefully how many certificates you create and what service you use, as there may be some rate limiting applied.
Let’s create a certificate for my cluster which is hosted.
# oc get certificate,certificaterequest,order
NAME READY SECRET AGE
certificate.cert-manager.io/cert-test-http01 True cert-test-dns01-b-sec 48m
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
certificaterequest.cert-manager.io/cert-test-http01 True True letsencrypt-prody system:serviceaccount:cert-manager:cert-manager 25m
NAME STATE AGE
order.acme.cert-manager.io/cert-test-http01-3937192702 valid 25m
Once the order switches from Pending to valid, your certificate is now available in the secret.
Get the certificate usinig the oc. You can also mount the secret or use the secret for the route
oc get secret cert-test-http01b-sec -oyaml
If you don’t have direct access to the internet, or the HTTP01 is not an option, you can use the cert-manager-webhook-ibmcis.
Clone the repository git clone https://github.com/IBM/cert-manager-webhook-ibmcis.git
Change to the directory cd cert-manager-webhook-ibmcis
Create the webhook project oc new-project cert-manager-webhook-ibmcis
Create the ibmcis deployment oc apply -f cert-manager-webhook-ibmcis.yaml
Once the pods are available and ready in the cert-manager-webhook-ibmcis, then we can proceed.
Create the api-token. It is recommended you use a service id with specific access to your CIS instance.
oc create secret generic ibmcis-credentials --from-literal=api-token="<YOUR API KEY>"
Retreive your CRN using the ibmcloud cli, and save the ID
❯ ibmcloud cis instances
Retrieving service instances for service 'internet-svcs'
OK
Name ID Location State Service Name
mycis crn:v1:bluemix:public:internet-svcs:global:a/<ACCOUNT_NUM>:<INSTANCE_ID>:: global active internet-svcs
Create the ClusterIssuer, updating YOUR_EMAIL and the CIS ID.
cat << EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prody
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: <YOUR_EMAIL>
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
webhook:
groupName: acme.borup.work
solverName: ibmcis
config:
apiKeySecretRef:
name: ibmcis-credentials
key: api-token
cisCRN:
- "crn:v1:bluemix:public:internet-svcs:global:a/<ACCOUNT_NUM>:<INSTANCE_ID>::"
EOF
The OpenShift Container Platform Multi-Arch Compute feature supports the pair of processor (ISA) architectures – ppc64le and amd64 in a cluster. With these pairs, there are various permutations when scheduling Pods. Fortunately, the platform has controls on where the work is scheduled in the cluster. One of these controls is called the node selector. This article outlines how to go about using Node Selectors at different levels – Pod, Project/Namespace, Cluster.
Consider the Pod definition for test, the nodeSelector has x: y and is matched with a Node which is labeled with .metadata.labels
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- name: test
image: "ocp-power.xyz/test:v0.0.1"
nodeSelector:
x: y
You can use node selectors on pods and labels on nodes to control where the pod is scheduled. With node selectors, OpenShift Container Platform schedules the pods on nodes that contain matching labels. To direct a Pod to a Power node, you could use the kubernetes.io/arch: ppc64le label.
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- name: test
image: "ocp-power.xyz/test:v0.0.1"
nodeSelector:
kubernetes.io/arch: ppc64le
You can see where the Pod is scheduled using oc get pods -owide.
❯ oc get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
test 1/1 Running 0 24d 10.130.2.9 mac-acca-worker-1 <none> <none>
You can confirm the architecture for each node oc get nodes mac-acca-worker-1 -owide. You’ll then see the uname is marked with ppc64le
❯ oc get nodes mac-acca-worker-1 -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
mac-acca-worker-1 Ready worker 25d v1.28.3+20a5764 192.168.200.11 <none> Red Hat Enterprise Linux CoreOS 414.92..... 5.14.0-284.41.1.el9_2.ppc64le cri-o://1.28.2-2.rhaos4.14.gite7be4e1.el9
This approach applies to high-level Kubernetes abstractions such as ReplicaSets, Deployments or DaemonSets.
Project / Namespace Level
Per OpenShift 4.14: Creating project-wide node selectors, the control of Pod creation may not be available in the Project or Namespace. This behavior leaves the customer without control over Pod placement. Kubernetes and OpenShift provide control over Pod placement when the control over the Pod definition is not possible.
Kubernetes enables this feature through the Namespace annotation scheduler.alpha.kubernetes.io/node-selector. You can read more about internal-behavior.
You can annotate the namespace:
oc annotate ns example scheduler.alpha.kubernetes.io/node-selector=kubernetes.io/arch=ppc64le
OpenShift enables this feature through Namespace annotation.
oc annotate ns example openshift.io/node-selector=kubernetes.io/arch=ppc64le
These direct the Pod to the right node architecture.
Cluster Level
Per OpenShift 4.14: Creating default cluster-wide node selectors, the control of Pod creation may not be available or there is a need for a default. This customer controls Pod placement through a default setting of the cluster-wide default node selector.
To configure the cluster-wide default, patch the Scheduler Operator custom resource (CR).
To direct scheduling to the other pair of architectures, you MUST define a nodeSelector to override the behavior.
Summary
You have seen how to control the distribution of work and how to schedule work with multiple architectures.
In a future blog, I’ll cover Multiarch Manager Operatorsource which aims to aims to address problems and usability issues encountered when working with Openshift clusters with multi-architecture compute nodes.
