Blog

  • Custom Metrics Autoscaling with KEDA: Reliable, Observable, and Ready for IBM Power

    Custom Metrics AutoScaler, Red Hat’s KEDA build, makes Kubernetes autoscaling more flexible, event-driven, and production-ready. With support for diverse scalers, fallback behavior, lifecycle events, and observability signals, CMA helps platform teams scale workloads based on the metrics that matter most to their applications.

    A key reliability improvement is KEDA’s fallback behavior. When an external metrics source becomes unavailable, CMA can preserve the last-known replica count instead of making unsafe scaling decisions. This behavior is validated for standard Deployments and also for Argo Rollouts, helping teams maintain stability during temporary metrics outages while supporting progressive delivery patterns. You should use the GitOps Operator to use with IBM Power and OpenShift.

    KEDA also improves event-driven visibility through the CloudEventSource custom resource. Scaling lifecycle events can be emitted as structured CloudEvents, including meaningful sourcesubject, and type fields. This makes it easier to integrate KEDA activity with event routers, audit systems, and operational workflows.

    The scaler ecosystem remains broad and practical. The cron scaler supports scheduled scale-up and scale-down windows, making it useful for predictable traffic patterns. The kubernetes-workload scaler enables one workload to scale based on the pod count of another workload, with activationValue thresholds helping avoid unnecessary scaling from low-signal activity.

    Resource-based autoscaling is also covered through CPU and memory scalers, which scale deployments when container utilization crosses configured percentage thresholds. These provide familiar autoscaling behavior while remaining part of KEDA’s unified scaling model.

    For event streaming workloads, the Kafka scaler supports scenarios such as consumer group lag, offset commit policies, and partition distribution strategies. This makes KEDA a strong fit for streaming systems where scale should reflect backlog and consumption pressure.

    For metrics-driven platforms, the Prometheus scaler allows deployments to scale from the result of a configurable PromQL query. This gives teams the freedom to scale from application, infrastructure, or business metrics already exposed through Prometheus.

    KEDA also strengthens operational confidence through observability. Tests validate that KEDA exports correct OpenTelemetry metrics to collectors and exposes well-formed Prometheus metrics from operator and adapter endpoints. These signals help teams monitor scaler health, adapter behavior, reconciliation activity, and autoscaling outcomes.

    Finally, Custom Metrics AutoScaler v2.19 adds support for IBM Power, expanding deployment options for organizations running Kubernetes on Power-based infrastructure.

    Together, these features make CMA a powerful Custom Metrics AutoScaler for modern Kubernetes environments: flexible in what it scales from, resilient when dependencies fail, observable by default, and increasingly portable across infrastructure platforms.

    Go forth and build with IBM Power and CMA. https://community.ibm.com/community/user/blogs/paul-bastide/2026/06/17/custom-metrics-autoscaler-operator-v2191-supports

  • Custom Metrics Autoscaler Operator v2.19.1 supports IBM Power

    Today, Red Hat released v2.19.1 of Custom Metrics AutoScaler. Based on KEDA, the optional operator increases or decreases the number of Pods, custom resource, or Job based on custom metrics; it’s not only about CPU and memory. You can control your applications to control usage.

    To get started:

    1. On OpenShift Container Platform web console, click Ecosystem > Installed Operators.
    2. Select Custom Metrics Autoscaler.
    3. On the Operator Details page, click the KedaController tab.
    4. On the KedaController tab, click Create KedaController and edit the file.
    apiVersion: keda.sh/v1alpha1
    kind: KedaController
    metadata:
      name: keda
      namespace: openshift-keda
    spec:
      metricsServer:
        logLevel: "0"
      operator:
        logEncoder: console
        logLevel: info
      serviceAccount: {}
      watchNamespace: ""
    

    You can integrate with Kafka

    helm repo add strimzi https://strimzi.io/charts/
    helm repo update
    helm upgrade --install --namespace strimzi --wait strimzi strimzi/strimzi-kafka-operator --version 0.47.0 --set watchAnyNamespace=true
    

    Good luck…

    References

    1. Red Hat OpenShift Docs: Chapter 3. Automatically scaling pods with the Custom Metrics Autoscaler Operator
    2. Red Hat Developer: Custom Metrics Autoscaler on OpenShift
    3. keda.sh: Kubernetes Event-driven Autoscaling
    4. Custom Metrics Autoscaler Operator v2.19.1 supports IBM Power
  • Using HTTPS-only Connections for Package Repositories

    In CentOS (by default), repositories use a metalink rather than a direct baseurl. The metalink is an endpoint that returns a list of geographic mirrors tailored to your server.

    While the metalink URL itself is usually secure (starting with https://), the parameters at the end of the URL explicitly tell the CentOS mirror infrastructure which protocols (http or https) your machine is willing to accept. If your network team enforces a strict “Deny All” outbound policy for port 80, your server will no longer mysteriously hang or fail when dnf attempts to route traffic through an HTTP mirror.

    If you take a look at your .repo files, you might spot “http” in our CentOS repository configurations, such as metalink=https://mirrors.centos.org/metalink?repo=centos-crb-source-$stream&arch=source&protocol=https,http Notice the trailing query parameter: protocol=https,http.

    To ensure that your package manager exclusively uses secure connections, we need to strip the ,http fallback from the protocol parameter across all repository files.

    You can do this quickly and safely using a single sed command, followed by clearing the local cache so dnf fetches a fresh, HTTPS-only mirror list.

    Run the following commands as root (or with sudo):

    # 1. Remove the HTTP fallback from all repo files
    sudo sed -i 's/protocol=https,http/protocol=https/g' /etc/yum.repos.d/*.repo
    
    # 2. Clear out the old DNF cache containing the insecure mirrors
    sudo dnf clean all
    
    # 3. Rebuild the cache with the new HTTPS-only rules
    sudo dnf makecache
    

    Now you can connect and use the https repository.

  • 2026-06: Additional IBM Power Open Source Images on the IBM Container Registry

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). These images are available for no-charge and can be used in your development and production environments.

    Image NameTag NameProject LicensesImage Pull CommandLast Published On
    kafka4.1.0-bvApache-2.0podman pull icr.io/ppc64le-oss/kafka-ppc64le:4.1.0-bvJune 10, 2026
    ansible-acme-test-container2.3.0GPL-3.0, Apache License 2.0podman pull icr.io/ppc64le-oss/ansible-acme-test-container-ppc64le:2.3.0June 10, 2026
    vllm0.9.1Apache-2.0podman pull icr.io/ppc64le-oss/vllm-ppc64le:0.9.1June 9, 2026

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/

  • How to Run Multiple Clusters with one bastion node

    To run multiple OpenShift clusters from one bastion requires managing dhcpd, named, http, haproxy with isolated configurations.

    After deploying with ocp4-upi-powervm, you can ‘move’ the configuration over

    1. dhcpd enables booting the rhcos nodes, which then can grab their configuration. dhcpd support include statements, allowing you to split subnets, host reservations, and cluster-specific configurations into separate files.

      1. Create the conf.d directory: mkdir -p /etc/dhcp/conf.d
      2. Modify your main /etc/dhcp/dhcpd.conf to include the directory. Add this at the bottom of the file: include "/etc/dhcp/conf.d/ocp-cluster-1.conf";
      3. Create the file /etc/dhcp/conf.d/ocp-cluster-1.conf – you’ll have to give the host unique names.
      subnet 10.20.176.0 netmask 255.255.240.0 {
      interface eth0;
          # Static entries
          host bootstrap { hardware ethernet fa:16:3e:ff:b7:b2; fixed-address 10.20.188.84; }
          host master-0 { hardware ethernet fa:16:3e:9b:c5:89; fixed-address 10.20.188.206; }
          host master-1 { hardware ethernet fa:16:3e:b7:ba:16; fixed-address 10.20.188.62; }
          host master-2 { hardware ethernet fa:16:3e:14:2c:ff; fixed-address 10.20.188.166; }
          host worker-0 { hardware ethernet fa:16:3e:97:7b:1b; fixed-address 10.20.188.79; }
          host worker-1 { hardware ethernet fa:16:3e:62:39:fe; fixed-address 10.20.188.234; }
          host worker-2 { hardware ethernet fa:16:3e:23:54:0a; fixed-address 10.20.188.131; }
          # this will not give out addresses to hosts not listed above
          #deny unknown-clients;
      
          # this is PXE specific
          filename "boot/grub2/powerpc-ieee1275/core.elf";
      
          next-server 10.20.188.128;
          }
      
      1. Restart the systemd service systemctl restart dhcpd
    2. If you are hosting ignition files on httpd on port 8080.

      1. Create the ignition folder mkdir -p /var/www/html/ignition/{ocp-cluster-1,ocp-cluster-2}
      2. Copy the ignition files into /var/www/html/ignition/ocp-cluster-#
      3. Or Download the ignitions curl -k -H "Accept: application/vnd.coreos.ignition+json;version=3.4.0" -o /var/www/html/ignition/power.ign https://api-int.XYZ.powervs-openshift-ipi.cis.ibm.net:22623/config/power
      4. Restore selinux restorecon -r /var/www/html/ignition
    3. HAProxy allows us to use separate use_backend and acl

      1. Edit /etc/haproxy/haproxy.cfg
      2. Add acl for the domain name based on hostname
      frontend https-all
      mode        tcp
      option      tcplog
      
      bind        *:443
      
      acl 02-https-ci req_ssl_sni -m end .mycluster1.ibm.net
      use_backend https-workers-02 if 02-https-ci
      
      acl 03-https req_ssl_sni -m end .mycluster2.ibm.net
      use_backend https-workers-03 if 03-https
      
      1. Create a backend target for the above:
      backend https-workers-03
      mode        tcp
      balance     roundrobin
      server      master1 192.168.3.11:443 check
      server      master2 192.168.3.12:443 check
      server      master3 192.168.3.13:443 check
      server      worker1 192.168.3.51:443 check
      server      worker2 192.168.3.52:443 check
      

    We use this approach in OCP LibVirt CI see haproxy_C155F2U31.cfg

    1. named support multiple conf files using the include directive

      1. Create the modular directory: mkdir -p /etc/named/conf.d
      2. Modify /etc/named.conf to include your custom zone files. include "/etc/named/conf.d/ocp-cluster-1.conf";
      3. Create the file /etc/named/conf.d/ocp-cluster-1.conf
      zone "mycluster2.ibm.net" IN {
          type master;
          file "/var/named/zones/db.ocp-cluster-1.local";
          allow-query { any; };
      };
      
      zone "122.168.192.in-addr.arpa" IN {
          type master;
          file "/var/named/zones/ocp-cluster-1.192.168.122";
          allow-query { any; };
      };
      

    Using this approach you’ll be able to share the bastion.

  • OpenShift Container Platform 4.22.0 has been released

    New features are:

    • Installer-provisioned infrastructure for IBM PowerVC is now generally available.
    • Enforce RSA key format for Installer-provisioned infrastructure on IBM Power® Virtual Server.
    • Harden the destroy logic for Installer-provisioned infrastructure on IBM Power® Virtual Server to simplify removing a cluster.
    • RHEL-10 Tech Preview with osImageStream

    Release Notes https://docs.redhat.com/en/documentation/openshift_container_platform/4.22/html/release_notes/ocp-4-22-release-notes#ocp-release-notes-ibm-power_release-notes 

    Video YouTube: What’s New in OpenShift 4.22 – Key Updates and New Features

  • R.reduce is not a function

    My teammate hit this issue `R.reduce is not a function` when using OperatorHub

    Workaround:

    oc patch operatorhubs/cluster --type merge --patch '{"spec":{"sources":[{"disabled": true,"name": "community-operators"},{"disabled": true,"name": "certified-operators}]}}'

    PR #16588 • Bug OCPBUGS-88027

    Thanks to https://access.redhat.com/solutions/7075189 for the solution

  • 2026-06: Additions IBM Power Open Source Images on the IBM Container Registry (Part 2)

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). These images are available for no-charge and can be used in your development and production environments.

    Image NameTag NameProject LicensesImage Pull CommandLast Published On
    envoy5.22.2Apache-2.0podman pull icr.io/ppc64le-oss/envoy-ppc64le:1.36.5June 8, 2026
    mongodb8.3.1Apache-2.0podman pull icr.io/ppc64le-oss/mongodb-ppc64le:8.3.1June 9, 2026
    kafka4.1.0-bvApache-2.0icr.io/ppc64le-oss/kafka-ppc64le:4.1.0-bvJune 10, 2026

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/

  • 2026-06: Additions IBM Power Open Source Images on the IBM Container Registry

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). These images are available for no-charge and can be used in your development and production environments.

    Image NameTag NameProject LicensesImage Pull CommandLast Published On
    grafana-operator-ppc64le5.22.2Apache-2.0podman pull icr.io/ppc64le-oss/grafana-operator-ppc64le:5.22.2June 5, 2026

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/

  • June 2026: Additions IBM Power Open Source Images on the IBM Container Registry

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). New images for redis-operator and opa (open policy agent) are particular interesting for those working with analytics and caching.

    redis-operator	v0.24.0 	Apache-2.0 	podman pull icr.io/ppc64le-oss/redis-operator-ppc64le:v0.24.0 	May 22, 2026
    opa-ppc64le 	v1.15.1 	Apache-2.0 	podman pull icr.io/ppc64le-oss/opa-ppc64le :v1.15.1 	May 29, 2026
    

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/