Blog

OpenShift… if you need a firewall

If your security posture requires a firewall, you can add it to your OpenShift cluster using the following:

Create a butane configuration

cat << EOF > 98-nftables-worker.bu
variant: openshift
version: 4.16.0
metadata:
  name: 98-nftables-worker
  labels:
    machineconfiguration.openshift.io/role: worker
systemd:
  units:
    - name: "nftables.service"
      enabled: true
      contents: |
        [Unit]
        Description=Netfilter Tables
        Documentation=man:nft(8)
        Wants=network-pre.target
        Before=network-pre.target
        [Service]
        Type=oneshot
        ProtectSystem=full
        ProtectHome=true
        ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf
        ExecReload=/sbin/nft -f /etc/sysconfig/nftables.conf
        ExecStop=/sbin/nft 'add table inet custom_table; delete table inet custom_table'
        RemainAfterExit=yes
        [Install]
        WantedBy=multi-user.target
storage:
  files:
  - path: /etc/sysconfig/nftables.conf
    mode: 0600
    overwrite: true
    contents:
      inline: |
        table inet custom_table
        delete table inet custom_table
        table inet custom_table {
            chain input {
                type filter hook input priority 0; policy accept;
                ip saddr 1.1.1.1/24 drop
            }
        }
EOF

Download butane

curl -o butane https://github.com/coreos/butane/releases/download/v0.23.0/butane-ppc64le-unknown-linux-gnu -L

Execute the butane

chmod +x butane; ./butane 98-nftables-worker.bu -o 98-nftables-worker.yaml

Run the nftables-worker.yaml butane

oc apply -f 98-nftables-worker.yaml

You can verify the workers drop the traffic.

Reference

OpenShift nftables.service https://access.redhat.com/articles/7090422

2025-04-25

Cool Feature… NodeDisruptionPolicies
I missed this feature in 4.17…. until I had to use it NodeDisruptionPolicies. If you are copying files over, you can avoid a MachineConfigPool reboot for files and services that depend on them. you can see more details Using node disruption policies to minimize disruption from machine config changes
```
apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
status:
  nodeDisruptionPolicyStatus:
    clusterPolicies:
      files:
      - actions:
        - type: None
        path: /etc/mco/internal-registry-pull-secret.json
```
Net… you can avoid a reboot when copying a file over/replacing a file and restarting a related service (already running).

FYI I ran across it with relations to nftables.service https://access.redhat.com/articles/7090422
2025-04-25
Red Hat OpenShift Container Platform on IBM Power Systems: Exploring Red Hat’s Multi-Arch Tuning Operator
The Red Hat Multi-Arch Tuning Operator optimizes workload placement within multi-architecture compute clusters. Pods run on the compute architecture for which the containers declare support. Where Operators, Deployments, ReplicaSets, Jobs, CronJob, Pods don’t declare a nodeAffinity, in most cases, the Pods that are generate are updated with the node affinity so it lands on the supported (declared) CPU Architecture.

For version 1.1.0, the Red Hat Multi-Arch Team, @Prashanth684, @aleskandro, @AnnaZivkovic and IBM Power Systems team @pkenchap have worked together to give cluster administrators better control and flexibility. The feature adds a plugins field in ClusterPodPlacementConfig and have build a first plugin called nodeAffinityScoring.

Per the docs, the nodeAffinityScoring plugin adds weights and influence to the scheduler with this process:
1. Analyzing the Pod’s containers for the supported architectures
2. Generate the Scheduling predicates for nodeAffinity, e.g., 75 weight on ppc64le
3. Filter out nodes that do not meet the Pod requirements, using the Predicates
4. Prioritizes the remaining nodes based on the architecture scores defined in the nodeAffinityScoring.platforms field.
To take advantages of this feature, use the following to asymmetrically load the Power nodes with work.
```
apiVersion: multiarch.openshift.io/v1beta1
kind: ClusterPodPlacementConfig
metadata:
  name: cluster
spec:
  logVerbosityLevel: Normal
  namespaceSelector:
    matchExpressions:
      - key: multiarch.openshift.io/exclude-pod-placement
        operator: Exists
  plugins:
    nodeAffinityScoring:
      enabled: true
      platforms:
        - architecture: ppc64le
          weight: 100
        - architecture: amd64
          weight: 50
```
Best wishes, and looking forward to hearing how you use the Multi-Arch Tuning Operator on IBM Power with Multi-Arch Compute.

References
2025-04-17

nx-gzip requires active_mem_expansion_capable

nx-gzip requires the licensed process caability active_mem_expansion_capable

for MACHINE in my-ranier1 my-ranier2
do
echo "MACHINE: ${MACHINE}"
for CAPABILITY in $(lssyscfg -r sys -F capabilities -m "${MACHINE}" | sed 's|,| |g' | sed 's|"||g')
do
echo "CAPABILITY: ${CAPABILITY}" | grep active_mem_expansion_capable
done
echo
done

The following shows:

MACHINE: my-ranier1
CAPABILITY: active_mem_expansion_capable
CAPABILITY: hardware_active_mem_expansion_capable
CAPABILITY: active_mem_mirroring_hypervisor_capable
CAPABILITY: cod_mem_capable
CAPABILITY: huge_page_mem_capable
CAPABILITY: persistent_mem_capable

MACHINE: my-ranier2
CAPABILITY: cod_mem_capable
CAPABILITY: huge_page_mem_capable
CAPABILITY: persistent_mem_capable

Then you should be all set to use nx-gzip on my-ranier1

Best wishes

2025-04-16

Setting up nx-gzip in a non-privileged container

In order to use nx-gzip on Power Systems with a non-privileged container, use the following recipe:

On each of the nodes, create the selinux power-nx-gzip.cil:

(block nx
    (blockinherit container)
    (allow process container_file_t ( chr_file ( map )))
)

Install the CIL on each worker node

sudo semodule -i power-nx-gzip.cil /usr/share/udica/templates/base_container.cil

I ran the following:

podman run -it --security-opt label=type:nx.process --device=/dev/crypto/nx-gzip registry.access.redhat.com/ubi9/ubi@sha256:a1804302f6f53e04cc1c6b20bc2204d5c9ae6e5a664174b38fbeeb30f7983d4e sh

I copied the files into the container using the container CONTAINER ID:

podman ps
podman cp temp 6a4d967f3b6b:/tmp
podman cp gzfht_test 6a4d967f3b6b:/tmp

Then running in the container:

sh-5.1# cd /tmp
sh-5.1# ./gzfht_test temp
file temp read, 1048576 bytes
compressed 1048576 to 1105922 bytes total, crc32 checksum = 3c56f054

You can use ausearch -m avc -ts recent | audit2allow to track down missing permissions

Hope this helps you…

Reference

https://github.com/libnxz/power-gzip

2025-04-09

Helpful Tool – mtr

I was not aware of the mtr which Network diagnostic tool combining 'traceroute' and 'ping'.

You can quickly install on RHEL/Centos with sudo dnf install -y mtr

The output is super helpful to where you have drops:

 mtr --report bastide.org
Start: 2025-04-04T12:44:43-0400
HOST: nx-gzip-d557-bastion-0.x Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.20.176.3                0.0%    10    1.9   1.2   0.9   1.9   0.3
  2.|-- 172.16.32.4                0.0%    10    0.7   0.7   0.7   0.8   0.0
  3.|-- att-vc-srx-interconnect.p  0.0%    10   30.2  33.9  25.4  62.6  11.0
  4.|-- XX.5.16.XXX                0.0%    10   11.8  11.8  11.7  12.0   0.1
  5.|-- po97.prv-leaf6a.net.unifi  0.0%    10   62.5  63.2  62.5  67.5   1.5

2025-04-04

DNS Resolver Hangs with OpenVPN
Running multiple OpenVPN on the mac, sometimes my DNS hangs and I can’t get the VPNs. I use this hack to get around it.
```
❯ sudo networksetup -setdnsservers Wi-Fi "Empty"
```
2025-04-03

Help… My Ingress is telling me OAuthServerRouteEndpointAccessibleControllerDegraded

My teammate hit an issue with Ingress Certificates not being valid:

oc get co ingress -oyaml
    message: |-
      OAuthServerRouteEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.mycluster.local/healthz": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-04-02T17:58:35Z is after 2025-02-13T20:04:16Z
      RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.mycluster.local] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.mycluster.local: x509: certificate has expired or is not yet valid: current time 2025-04-02T17:58:33Z is after 2025-02-13T20:04:16Z

The Red Hat docs and tech articles are great. I found How to redeploy/renew an expired default ingress certificate in RHOCP4?

I ran the following on a non-production cluster:

Renewed the ingress CA:

oc get secret router-ca -oyaml -n openshift-ingress-operator> router-ca-2025-04-02.yaml
oc delete secret router-ca -n openshift-ingress-operator
oc delete pod --all -n openshift-ingress-operator
wait 30
oc get secret router-ca -n openshift-ingress-operator
oc get po -n openshift-ingress-operator

Recreate the wild-card ingress certificate using the new ingress CA:

oc get secret router-certs-default -o yaml -n openshift-ingress > router-certs-default-2025-04-02.yaml
oc delete secret router-certs-default -n openshift-ingress 
oc delete pod --all -n openshift-ingress 
wait 30
oc get secret router-certs-default -n openshift-ingress 
oc get po -n openshift-ingress

Checked the ingress

curl -v https://oauth-openshift.apps.mycluster.local/healthz -k
*  subject: CN=*.apps.mycluster.local
*  start date: Apr  2 19:08:33 2025 GMT
*  expire date: Apr  2 19:08:34 2027 GMT

Update ca-trust

oc -n openshift-ingress-operator get secret router-ca -o jsonpath="{ .data.tls\.crt }" | base64 -d -i > ingress-ca-2025-04-02.crt
cp /root/ingress-ca-2025-04-02.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust

oc login -u kubeadmin -p YOUR_PASSWORD https://api.mycluster.local:6443

You’ve seen how to recreate the cert.

You should use the cert-manager operator from Red Hat.

2025-04-02

Multi-Arch Tuning Operator 1.1.0 Released
The Red Hat team has released a new version of the Multi-Arch Tuning Operator.

In Multi-Arch Compute clusters, the Multiarch Tuning Operator influences the scheduling of Pods, so application run on the supported architecture.

You can learn more about it at https://catalog.redhat.com/software/containers/multiarch-tuning/multiarch-tuning-operator-bundle/661659e9c5bced223a7f7244

Addendum

My colleague, Punith, worked with the Red Hat team to add NodeAffinityScoring and plugin support to the Multi-Arch Tuning Operator and ClusterPodPlacementConfig. This feature allows users to define cluster-wide preferences for specific architectures, influencing how the Kubernetes scheduler places pods. It helps optimize workload distribution based on preferred node architecture.
```
	Spec:
	    Plugins:
		NodeAffinityScoring:
		   enabled: true
		   platforms:
		   - architecture: ppc64le
		     weight: 100
		   - architecture: amd64
		     weight: 50
```
2025-03-18
FIPS support in Go 1.24

Kudos to the Red Hat team. link

The benefits of native FIPS support in Go 1.24

The introduction of the FIPS Cryptographic Module in Go 1.24 marks a watershed moment for the language’s security capabilities. This new module provides FIPS 140-3-compliant implementations of cryptographic algorithms, seamlessly integrated into the standard library. What makes this particularly noteworthy is its transparent implementation. Existing Go applications can leverage FIPS-compliant cryptography without requiring code changes.

Build-time configuration through the GOFIPS140 environment variable, allowing developers to select specific versions of the Go Cryptographic Module.

GOFIPS140=true go build

Runtime control via the fips140 GODEBUG setting, enabling dynamic FIPS mode activation.

GODEBUG=

Keep these in your toolbox along with GOARCH=ppc64le

2025-03-17

Blog

Reference

References

The benefits of native FIPS support in Go 1.24