Blog

  • CEL Architecture Placement in the Multi-Architecture Tuning Operator

    Author: Pranita Turrey/IBM, Paul Bastide/IBM, Punith Kenchappa/IBM

    Audience: Cluster administrators responsible for workload placement on heterogeneous OpenShift clusters.

    This Blog explains how to configure and operate Common Expression Language (CEL) Architecture Placement using the Multi-Architecture Tuning Operator (MTO). It covers the concepts you need to understand, the configuration you need to create, and the outcomes you can expect. Implementation internals, detailed field references, and product-specific examples are collected in appendices at the end.



    1. Why CEL Architecture Placement

    The Multi-Architecture Tuning Operator (MTO) automates architecture-aware Pod scheduling on heterogeneous clusters. By default, MTO inspects a Pod’s container images and sets node affinity so the Pod runs only on nodes whose CPU architecture supports those images. This is called image-based placement.

    Image-based placement covers the common case well. It has a structural limitation, though: when an image is built for multiple architectures, inspection alone cannot choose between them. If your cluster has both amd64 and ppc64le nodes and a workload’s image supports both, image-based placement will declare both eligible—and the scheduler will place the Pod wherever capacity is available.

    CEL Architecture Placement closes this gap. By writing a small set of rules in a PodPlacementConfig, you can match Pods by their existing labels, annotations, or name patterns and assign them a specific architecture. The result:

    • Workloads land where they belong—on the right architecture for performance, licensing, or operational policy—without touching a single Deployment manifest.
    • Migrations are incremental—move a component at a time by flipping a label, not by editing dozens of YAML files.
    • Multi-architecture compute environments stay organized—different components in the same namespace can have distinct placement rules enforced automatically.

    Prerequisite: A ClusterPodPlacementConfig named cluster must exist before any PodPlacementConfig can be created. The admission webhook enforces this.

    Business value

    CEL Architecture Placement enables teams to enforce architecture placement policy without modifying application manifests. Placement is expressed declaratively, version-controlled alongside infrastructure configuration, and applied consistently across all Pods in a namespace from the moment of creation.

    This has direct operational value:

    • Reduced toil during migrations: moving a component to a new architecture requires only a label change and a rule update—not Deployment restructuring.
    • Auditable policy: placement decisions are inspectable in the PodPlacementConfig resource, not buried in Deployment YAML across dozens of namespaces.

    When to use CEL Architecture Placement

    Use CEL Architecture Placement when:

    • Your cluster has nodes of multiple architectures and you need workloads to land on specific ones.
    • Multi-arch images make image-based placement insufficient (MTO cannot choose between equally-supported architectures).
    • You want to enforce placement policy at the cluster level without requiring changes to application team manifests.
    • You are migrating workloads incrementally from one architecture to another.
    • You need different components in the same namespace to run on different architectures.

    Continue using image-based placement (MTO’s default) when:

    • All workloads in a namespace should simply run on architectures their images support, with no explicit routing policy.
    • You have no preference between architectures and want the scheduler to decide.

    2. Core MTO Concepts

    This section defines the terms used throughout this document. No prior knowledge of MTO is required.

    TermMeaning
    Multi-Architecture Tuning Operator (MTO)A Kubernetes operator that automates architecture-aware Pod scheduling on clusters with nodes of multiple CPU architectures (e.g., amd64 and ppc64le).
    ClusterPodPlacementConfigA cluster-scoped resource that activates MTO for the cluster. Required before any namespace-level configuration can be created. Only one exists, named cluster.
    PodPlacementConfigA namespace-scoped resource that configures placement rules for Pods in that namespace. Multiple resources can coexist; they are evaluated in priority order.
    CEL Architecture PlacementThe placement mode described in this document. You write CEL rules inside a PodPlacementConfig to explicitly assign Pods to architectures based on their metadata.
    Image-based placementMTO’s default mode: inspect container image manifests to determine which architectures the Pod supports. CEL placement bypasses this entirely for matched Pods.
    fallbackArchitecturesThe architectures applied when CEL placement is enabled but no rule matches. Required field. Think of it as “everything else runs here.”
    Scheduling gateA mechanism that holds a Pod in a pending state until MTO has finished evaluating it and applied the correct architecture constraint. Once MTO is done, the gate is removed and the scheduler takes over. Administrators observe this as a brief pending state on new Pods; it is expected behavior.

    Note: CEL placement sets required (hard) node affinity. The NodeAffinityScoring plugin sets preferred (soft) node affinity. Both can coexist in the same PodPlacementConfig without conflict.


    3. How CEL Placement Works — Operational View

    When a Pod is created in a namespace where a PodPlacementConfig with the CEL placement plugin enabled exists, MTO intercepts it and applies an architecture constraint before the Pod reaches the scheduler. From an administrator’s perspective, the key outcomes are:

    • Architecture is enforced automatically. You do not need to modify Deployment manifests. Where a CEL expression matches, its configured architecture is applied; if no rule matches, fallbackArchitectures is used.
    • Pods briefly appear pending. MTO holds each Pod briefly with a scheduling gate while evaluating rules and applying the constraint. This is expected behavior.
    • The result is required node affinity. The Pod’s spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution is set to the target architecture. The scheduler will only place the Pod on a node matching that architecture.
    • Existing architecture constraints are overridden. Any prior kubernetes.io/arch node selector or architecture affinity on the Pod is replaced by the CEL-derived value. Non-architecture constraints (zone, OS, pod affinity, preferred affinity) are left untouched.

    Decision flow

    No
    
    Yes
    
    Rule matches
    
    No rule matches
    
    All rules error
    
    CEL constraint applies
    
    None applies
    
    Pod submitted
    
    Applicable PodPlacementConfig?
    
    Image-based detection / cluster fallback
    
    Evaluate rules in order
    
    Apply architecture
    
    Apply fallbackArchitectures
    
    Try lower-priority PodPlacementConfig
    
    

    For full precedence rules—including multiple PodPlacementConfig resources, image-based detection, and cluster-level fallback—see Section 7.

    For details on how MTO implements this internally (admission webhook vs. reconciliation loop, immutability constraints), see Appendix C.


    4. Configuring CEL Architecture Placement

    Step 1: Enable MTO for the cluster

    A ClusterPodPlacementConfig named cluster must exist. If it does not already exist in your cluster:

    apiVersion: multiarch.openshift.io/v1beta1
    kind: ClusterPodPlacementConfig
    metadata:
      name: cluster
    spec: {}
    
    oc apply -f cluster-config.yaml
    

    Step 2: Create a PodPlacementConfig in your namespace

    A minimal PodPlacementConfig with CEL placement enabled:

    apiVersion: multiarch.openshift.io/v1beta1
    kind: PodPlacementConfig
    metadata:
      name: my-placement-rules
      namespace: my-namespace
    spec:
      priority: 10
      plugins:
        celArchitecturePlacement:
          enabled: true
          fallbackArchitectures:
            - s390x             # applied when no rule matches; required
          rules:
            - name: postgres-on-ppc64le
              expression: |
                'app.kubernetes.io/component' in self.metadata.labels &&
                self.metadata.labels['app.kubernetes.io/component'] == 'database'
              architectures:
                - ppc64le
    

    Key points:

    • fallbackArchitectures is required. Set it to the architecture that should apply to all Pods that do not match a specific rule.
    • priority must be unique within the namespace. Values range from 0 to 255. Higher values are evaluated first.
    • labelSelector is optional. When omitted, the config applies to every Pod in the namespace.
    • rules is optional. A config with no rules applies fallbackArchitectures to every matched Pod—useful for pinning an entire namespace to a single architecture.

    labelSelector scopes which Pods this PodPlacementConfig considers at all. It is a standard Kubernetes label selector evaluated against the Pod’s own labels. When set, only Pods whose labels match are subject to this config’s rules and fallback. Omit it to apply the config to all Pods in the namespace. See the advanced configuration reference for examples combining labelSelector with CEL rules.

    For a complete field reference, see the advanced configuration reference.


    5. Writing Maintainable CEL Rules

    Each rule’s expression is evaluated against a restricted view of the Pod. The self variable exposes the Pod’s metadata only:

    CEL pathTypeTypical use
    self.metadata.namestringMatch a specific, known Pod name
    self.metadata.generateNamestringMatch Pods from a named Deployment or StatefulSet
    self.metadata.namespacestringNamespace of the Pod
    self.metadata.labelsmap<string, string>Match by workload label
    self.metadata.annotationsmap<string, string>Match by operator-set annotation

    self.spec and self.status are not available. Expressions that reference them are rejected when the PodPlacementConfig is saved, not at runtime.

    Syntax examples

    Match Pods created by a Deployment named worker (controller-managed Pods use generateName):

    self.metadata.generateName.startsWith('worker-')
    

    Match by a simple label value:

    has(self.metadata.labels.tier) && self.metadata.labels.tier == 'backend'
    

    Match by a label key containing dots or slashes (use bracket notation):

    'app.kubernetes.io/component' in self.metadata.labels &&
    self.metadata.labels['app.kubernetes.io/component'] == 'database'
    

    Match by annotation (useful for operator-managed workloads):

    'config.company.io/placement' in self.metadata.annotations &&
    self.metadata.annotations['config.company.io/placement'] == 'ppc64le'
    

    Catch-all rule (matches every Pod — useful as a final rule to make fallback intent explicit):

    true
    

    Label access: dot vs. bracket notation

    Key typeExistence checkValue check
    Simple alphanumeric (tier, app)has(self.metadata.labels.tier)self.metadata.labels.tier == 'value'
    Contains dots or slashes (app.kubernetes.io/component)'key' in self.metadata.labelsself.metadata.labels['key'] == 'value'

    Missing keys are treated as a non-match. MTO handles missing-key evaluation errors as a non-match, allowing evaluation to continue to the next rule or fallbackArchitectures. You can safely check for a label’s presence before reading its value.

    What is rejected at save time

    Expressions that do not compile or violate the allowed schema are rejected when the PodPlacementConfig is saved. They can never reach runtime.

    ProblemExample
    Non-boolean resultself.metadata.name (returns a string)
    Syntax errorself.metadata.name == (incomplete)
    Unavailable fieldself.spec.nodeName == 'worker-1'
    Typo in field pathself.metadata.labells.app

    Maintainability note

    CEL expressions are concise and powerful, but they can become difficult to read as rules accumulate. Some practices that help:

    • Give each rule a descriptive name that explains its intent, not just its mechanics.
    • Use the labelSelector on the PodPlacementConfig to narrow scope before writing rules—it reduces the surface area each rule needs to cover.
    • Prefer label-based matching over generateName-based matching where possible; labels are more stable and easier to reason about over time.
    • Keep the number of rules in a single config small. If rules are growing unwieldy, consider whether separate PodPlacementConfig resources with distinct priorities would be clearer.

    A follow-up article on CEL expression maintainability, testing patterns, and advanced rule organization is forthcoming.


    6. Rule Evaluation and Fallback Behavior

    This section is the single authoritative reference for how rules are evaluated and what happens when no rule matches.

    Within a single PodPlacementConfig

    Rules are evaluated in list order. The first rule that evaluates to true wins; remaining rules are not evaluated. If no rule matches, fallbackArchitectures is applied.

    If a rule’s expression fails at runtime (unexpected state, transient error), that rule is treated as a non-match and evaluation moves to the next rule. A single broken rule does not block the others.

    Important exception: if every rule in a PodPlacementConfig fails at runtime, the entire config is skipped and its fallback is not applied. Lower-priority configs or image-based detection may then handle the Pod. See Section 8.

    Across multiple PodPlacementConfigs

    When more than one PodPlacementConfig applies to a Pod, they are evaluated in descending priority order; the first config that successfully applies a constraint wins. For the full precedence rules—including the priority cascade, runtime error behaviour, and interaction with image-based detection—see Section 7.

    Fallback behavior

    fallbackArchitectures is applied when the plugin is enabled and no rule matches—including when no rules are configured. It is a required field.

    Recommended pattern: set fallbackArchitectures to the architecture that the majority of Pods in the namespace should use. Write explicit rules only for the exceptions. For example: if 90% of workloads should run on amd64 and a small subset must run on ppc64le, set fallbackArchitectures: [amd64] and write rules only for the ppc64le workloads.


    7. Precedence and Interaction with Other Placement Methods

    This section is the single authoritative reference for how CEL placement interacts with MTO’s image-based path, fallback architectures, and other placement constraints.

    CEL placement takes precedence over image-based placement. When the celArchitecturePlacement plugin is enabled and applies to a Pod—whether via a rule match or the fallback—image-based detection is bypassed entirely for that Pod. MTO does not inspect container images and does not consult the cluster-level fallbackArchitecture from ClusterPodPlacementConfig.

    Image-based placement only runs when no matching PodPlacementConfig applies a CEL constraint:

    ConditionArchitecture source
    CEL plugin enabled; rule matchesCEL rule’s architectures
    CEL plugin enabled; no rule matchesfallbackArchitectures
    CEL plugin disabledImage-based detection (or cluster fallback)
    CEL plugin enabled; all rules error; no lower-priority config appliesImage-based detection (or cluster fallback)
    No matching PodPlacementConfig existsImage-based detection (or cluster fallback)

    Existing architecture constraints on the Pod are overridden. When the plugin applies, any kubernetes.io/arch entry in nodeSelector and any architecture term in required node affinity is removed and replaced with the CEL-derived value.

    Other constraints are preserved. Zone affinity, OS selectors, pod affinity, and preferred affinity (NodeAffinityScoring) are left untouched. CEL placement and NodeAffinityScoring can coexist in the same PodPlacementConfig without conflict.

    Priority cascade

    Yes
    
    No: all rules error
    
    
    
    Yes
    
    No: no lower-priority config applies
    
    Pod submitted
    
    Highest-priority applicable PodPlacementConfig
    
    CEL constraint successfully applied?
    
    Architecture applied
    
    Next lower-priority applicable PodPlacementConfig
    
    CEL constraint successfully applied?
    
    Image-based detection / cluster fallback
    
    

    8. Error Handling

    MTO combines configuration-time validation with runtime error handling to isolate expression errors and allow rule evaluation to continue where possible.

    Runtime and admission error handling

    BehaviorHow it is enforced
    Invalid expressions rejected at configuration timeExpressions are compiled against the allowed schema when the PodPlacementConfig is saved. Invalid, non-boolean, or schema-violating expressions are rejected immediately.
    Runtime rule errorThat rule is skipped; evaluation continues with the next rule. If every rule fails, the entire config is skipped (fallback not applied) and lower-priority configs or image-based detection handle the Pod. See Section 7 for the full cascade.

    Configuration-time errors

    These errors are caught when the PodPlacementConfig is saved, not at runtime:

    ProblemError message
    Invalid CEL syntaxinvalid CEL expression in rule "…": CEL compilation error: Syntax error …
    Non-boolean expressioninvalid CEL expression in rule "…": CEL expression must return a boolean, got …
    Reference to self.spec.* or self.status.*invalid CEL expression in rule "…": … field 'spec'/'status' …
    Typo in a metadata fieldCompilation fails against the typed schema

    Operational note on broad rules: a rule expression: true with no labelSelector on the config matches every Pod in the namespace. This may be intentional (a namespace-wide architecture pin), but confirm the intended scope using the labelSelector on the PodPlacementConfig to avoid unintended placement. CEL placement controls where a Pod is scheduled but does not verify that the Pod’s images support that architecture. If you assign a workload to an architecture its image does not support, the Pod can fail to pull its image. Always verify image compatibility before applying new placement rules in production. Use oc image info <image> or podman manifest inspect <image> to check multi-arch support.

    Account for improperly specified Pods. Pods that are missing expected labels or annotations will not match label- or annotation-based rules and will fall through to fallbackArchitectures. Ensure your fallback is appropriate for unlabelled or misconfigured Pods, and consider adding a monitoring check for Pods landing on the fallback architecture unexpectedly.


    9. End-to-End Administrator Example

    This example shows the complete flow for the most common operational pattern: default most workloads to one architecture, and explicitly route a small set of workloads to another.

    This is a general pattern that any administrator can adapt. For a product-specific example using this same pattern in an IBM Cloud Pak for Data / Watsonx.data namespace, see Appendix B.

    Scenario

    A platform team runs a multi-architecture amd64/ppc64le/s390x cluster. The analytics-ns namespace hosts several workloads:

    • An analytics engine (component: engine) that must run on ppc64le for performance.
    • A monitoring agent (app: monitoring) that must run on s390x via an explicit rule.
    • A frontend UI and everything else defaults to amd64 via fallback.

    All workloads use multi-arch images that technically support multiple architectures. Image inspection alone cannot enforce this placement policy.

    Step 1 — Ensure a ClusterPodPlacementConfig exists

    apiVersion: multiarch.openshift.io/v1beta1
    kind: ClusterPodPlacementConfig
    metadata:
      name: cluster
    spec: {}
    
    oc apply -f cluster-config.yaml
    

    Step 2 — Create the PodPlacementConfig

    apiVersion: multiarch.openshift.io/v1beta1
    kind: PodPlacementConfig
    metadata:
      name: two-tier-placement
      namespace: analytics-ns
    spec:
      priority: 10
      plugins:
        celArchitecturePlacement:
          enabled: true
          fallbackArchitectures:
            - amd64              # everything that doesn't match a rule lands here
          rules:
            - name: engine-on-ppc64le
              expression: |
                'component' in self.metadata.labels &&
                self.metadata.labels['component'] == 'engine'
              architectures:
                - ppc64le
            - name: monitoring-agent-on-s390x
              expression: |
                'app' in self.metadata.labels &&
                self.metadata.labels['app'] == 'monitoring'
              architectures:
                - s390x
    
    oc apply -f two-tier-placement.yaml
    oc get podplacementconfig -n analytics-ns
    

    Step 3 — Deploy workloads and observe placement

    Analytics engine Pod (component: engine):

    apiVersion: v1
    kind: Pod
    metadata:
      name: analytics-engine
      namespace: analytics-ns
      labels:
        component: engine
    spec:
      containers:
      - name: engine
        image: quay.io/example/analytics-engine:latest
    

    Rule engine-on-ppc64le matches → ppc64le is applied.

    Frontend Pod (component: frontend):

    apiVersion: v1
    kind: Pod
    metadata:
      name: analytics-ui
      namespace: analytics-ns
      labels:
        component: frontend
    spec:
      containers:
      - name: ui
        image: quay.io/example/analytics-ui:latest
    

    No rule matches → fallback amd64 is applied.

    Monitoring agent (app: monitoring):

    apiVersion: v1
    kind: Pod
    metadata:
      name: monitoring-agent
      namespace: analytics-ns
      labels:
        app: monitoring
    spec:
      containers:
      - name: agent
        image: quay.io/example/monitoring:latest
    

    Rule monitoring-agent-on-s390x matches → s390x is applied.

    Step 4 — Validate the placement

    After MTO processes each Pod, verify the resulting node affinity:

    oc get pod analytics-engine -n analytics-ns -o jsonpath='{.spec.affinity}'
    

    Expected output for analytics-engine (ppc64le via rule match):

    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - ppc64le
    

    For analytics-ui, the same structure will show values: [amd64] (via fallback). For monitoring-agent, it will show values: [s390x] (via rule match).

    What to check:

    • Each Pod’s spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution reflects the expected architecture.
    • Pods are in Running state (not stuck in Pending beyond the brief scheduling gate window).
    • Pod events do not show image pull errors (which would indicate an architecture/image mismatch).

    10. Conclusion

    CEL Architecture Placement gives cluster administrators direct, policy-driven control over workload placement in heterogeneous clusters—without requiring application teams to modify their Deployment manifests.

    For operations teams, this reduces toil during architecture migrations and provides clear, auditable placement policy expressed as code. A single PodPlacementConfig can govern an entire namespace, routing each workload to the right architecture automatically.

    For platform engineers, this makes multi-architecture compute composable: multiple PodPlacementConfig resources can coexist in a namespace with distinct priorities, allowing team-level and platform-level placement policies to stack cleanly.

    Adopt CEL Architecture Placement when you need explicit, policy-driven control over which architectures workloads land on—especially in multi-architecture compute namespaces where image-based detection alone is insufficient. Start with the default-to-one-architecture-with-exceptions pattern, validate in a non-production namespace first, and extend incrementally.


    Appendix A: Placement Strategies and Design Patterns

    Recommended adoption patterns

    Before applying placement rules in production, follow these steps:

    1. Start in a non-production namespace. Apply your PodPlacementConfig and verify that rules match the intended Pods before rolling out to production.
    2. Default to one architecture; route exceptions explicitly. Set fallbackArchitectures to the majority architecture and write rules only for the minority. This is the most maintainable long-term pattern. See Pattern 1 below.
    3. Use labels as the placement control plane. Instead of writing rules that match on Pod name or generateName, have your CI/CD pipeline apply a placement label (e.g., arch-placement: ppc64le) to the Pod template. Rules then stay stable as workload names change. See Pattern 3 and Pattern 4 for annotation- and name-based alternatives.
    4. Layer configs by concern. Use multiple PodPlacementConfig resources with distinct priorities to separate platform-level policy (high priority) from team-level exceptions (lower priority).

    Pattern 1: Default to one architecture; explicitly route exceptions

    The most common pattern. Set fallbackArchitectures to the majority architecture and write rules only for workloads that need something different. Rules are evaluated top-to-bottom; the first match wins. Everything else falls through to the fallback.

    fallbackArchitectures:
      - amd64
    rules:
      - name: high-memory-workload-on-ppc64le
        expression: |
          'workload-class' in self.metadata.labels &&
          self.metadata.labels['workload-class'] == 'high-memory'
        architectures:
          - ppc64le
    

    Effect: every Pod in the namespace lands on amd64 unless it carries workload-class: high-memory, in which case it lands on ppc64le. This is also the pattern used in the Watsonx.data example (Appendix B).

    Pattern 2: Incremental architecture migration using a label gate

    A team migrating workloads from amd64 to ppc64le adds a label (migration-ready: "true") to each component as it is validated on the new architecture. No Deployment manifest restructuring is required.

    fallbackArchitectures:
      - amd64
    rules:
      - name: migrated-workloads-on-ppc64le
        expression: |
          'migration-ready' in self.metadata.labels &&
          self.metadata.labels['migration-ready'] == 'true'
        architectures:
          - ppc64le
    

    To move a component, apply the label to its Pod template. To roll back, remove the label. The PodPlacementConfig does not change.

    Pattern 3: Placement via annotation for operator-managed workloads

    When Pods are created by operators that set annotations but not labels, use annotation-based matching:

    rules:
      - name: annotated-ppc64le-workloads
        expression: |
          'platform.company.io/arch-preference' in self.metadata.annotations &&
          self.metadata.annotations['platform.company.io/arch-preference'] == 'ppc64le'
        architectures:
          - ppc64le
    

    Pattern 4: Matching controller-managed Pods by name prefix

    Pods created by a Deployment named my-service will have generateName: my-service-. Use generateName to match them reliably when label-based matching is not available:

    rules:
      - name: my-service-on-ppc64le
        expression: self.metadata.generateName.startsWith('my-service-')
        architectures:
          - ppc64le
    

    Prefer label-based matching where possible. generateName matching is sensitive to Deployment rename operations and may require rule updates if workload names change.

    CEL expression best practices, testing patterns, and maintainability guidance are covered in a forthcoming follow-up article.


    Appendix B: Watsonx.data / cpd-instance Example

    This appendix documents a validated configuration for a namespace hosting IBM Cloud Pak for Data (CPD) and Watsonx.data workloads on a multi-architecture amd64/ppc64le cluster. It is provided as a reference for the general pattern described in Appendix A, Pattern 1: a namespace where a small number of components are qualified on ppc64le and everything else defaults to amd64.

    Applicability

    This configuration applies to any platform product where:

    • Most workloads can run on the default architecture (amd64).
    • A specific, known set of components has been tested and qualified on a secondary architecture (ppc64le).
    • Placement rules should be scoped to the product’s namespace to avoid cross-namespace interference.

    Configuration

    apiVersion: multiarch.openshift.io/v1beta1
    kind: PodPlacementConfig
    metadata:
      name: cpd-cel-placement
      namespace: cpd-instance
    spec:
      priority: 200
      # No labelSelector — this config applies to all pods in cpd-instance.
      # The CEL rules below determine per-workload architecture.
      plugins:
        celArchitecturePlacement:
          enabled: true
          fallbackArchitectures:
            - amd64              # all unmatched workloads run on amd64
          rules:
            - name: ibm-nginx-on-ppc64le
              expression: |
                'component' in self.metadata.labels &&
                self.metadata.labels['component'] == 'ibm-nginx'
              architectures:
                - ppc64le
            - name: runtime-manager-api-on-ppc64le
              expression: |
                'component' in self.metadata.labels &&
                self.metadata.labels['component'] == 'runtime-manager-api'
              architectures:
                - ppc64le
            - name: cpd-mgmt-on-ppc64le
              expression: |
                'component' in self.metadata.labels &&
                self.metadata.labels['component'] == 'cpd-mgmt'
              architectures:
                - ppc64le
    

    How it works

    The config has no labelSelector, so it applies to every Pod in cpd-instance. Three rules match the specific CPD components that are qualified on ppc64le; all other Pods fall through to the amd64 fallback.

    Pod created in cpd-instance
          │
          ├── component == 'ibm-nginx'           → ppc64le
          ├── component == 'runtime-manager-api' → ppc64le
          ├── component == 'cpd-mgmt'            → ppc64le
          └── (anything else)                    → amd64 (fallback)
    

    Deploying

    oc apply -f cpd-cel-placement.yaml
    oc get podplacementconfig -n cpd-instance
    

    Image compatibility

    This config assigns ppc64le to specific components. Before applying it in production, confirm that every container image used by those components (ibm-nginx, runtime-manager-api, cpd-mgmt) includes a ppc64le manifest. Assigning a workload to ppc64le when its image has no ppc64le layer can cause the Pod to fail to pull its image.

    oc image info <image>
    # or
    podman manifest inspect <image>
    

    Appendix C: Advanced Configuration Reference

    The detailed configuration reference — including the complete PodPlacementConfig field table, the full self variable schema, internal implementation details, and labelSelector usage patterns — has been moved to a dedicated document:

    CEL Architecture Placement: Advanced Configuration Reference


    Appendix D: Testing, Validation Results, and Lessons Learned

    The CEL architecture placement feature is covered by unit tests, admission webhook tests, and controller integration tests (using envtest).

    Test coverage includes:

    • CEL compilation and runtime evaluation for all documented expression patterns
    • Admission-time rejection of invalid, non-boolean, and schema-violating expressions
    • Rule ordering and first-match semantics
    • Fallback behavior when no rule matches
    • Priority-based selection across multiple PodPlacementConfig resources
    • Removal of existing architecture constraints from nodeSelector and required node affinity
    • Preservation of non-architecture constraints
    • Idempotent re-reconciliation
    • Coexistence with NodeAffinityScoring
    • The all-errored-rules bypass behavior (config skipped entirely if every rule fails at runtime)
  • 2026-10: Additions IBM Power Open Source Images on the IBM Container Registry

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). These images are available for no-charge and can be used in your development and production environments.

    Image NameTag NameProject LicensesImage Pull CommandLast Published On
    ansible-cloudstack-test-container1.7.0Apache License 2.0podman pull icr.io/ppc64le-oss/ansible-cloudstack-test-container-ppc64le:1.7.02026-10-14
    docling-serve1.35.0MITpodman pull icr.io/ppc64le-oss/docling-serve-ppc64le:1.35.010/14/2026
    ansible-cloudstack-test-container1.7.0Apache License 2.0podman pull icr.io/ppc64le-oss/ansible-cloudstack-test-container-ppc64le:1.7.02026-10-14
    pgvector0.8.6PostgreSQLpodman pull icr.io/ppc64le-oss/pgvector-ppc64le:0.8.62026-09-03
    pulsarv4.2.3Apache-2.0podman pull icr.io/ppc64le-oss/pulsar-ppc64le:4.2.32026-09-01

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/

  • Diagnosing a Pod Crash on OpenShift: SIGSEGV vs OOMKill

    When a your pod crashes with a segmentation fault signal near address 0x0 or 0x20, that points to an invalid pointer dereference — not an out-of-memory kill. The two failure modes look similar on the surface but require different investigation paths.

    1. Confirm the Actual Exit Cause
    oc get pod <pod> -n <namespace> -o json \
      | jq '.status.containerStatuses[]'
    
    SignalExit CodeMeaning
    SIGSEGV139 / signal 11Segmentation fault → investigate the crash
    —137 / OOMKilledOut of memory → review resource limits instead

    If you see OOMKilled, stop here and adjust the pod’s memory limits. If you see exit code 139, continue below.

    1. Open a Debug Shell on the Affected Node
    oc debug node/<node>
    

    Inside the debug shell:

    chroot /host
    coredumpctl list
    coredumpctl info <host-pid>
    

    Match the entry by crash timestamp and executable name. Use the PID reported by coredumpctl — not the container’s PID 1.

    1. Export the Core Dump

    Still inside the debug shell, write the dump to a location with sufficient free space:

    umask 077
    coredumpctl --output=/var/tmp/app.core dump <host-pid>
    
    1. Copy the Dump Locally

    From a second terminal (keep the debug session open), copy the file using the debug pod’s name:

    oc cp -n <debug-namespace> \
      <debug-pod>:/host/var/tmp/app.core ./app.core
    

    Next Steps

    Hand the app.core file off to the engineering team for analysis. A kernel dump is unlikely to add value here — the core dump from the process itself is what matters for a near-zero pointer fault.

  • Great News – New Release of Security Profiles Operator – v1.1.0

    Security Profiles Operator v1.1.0 is now available, bringing AppArmor network policy support, operator health probes, webhook tolerations, tighter SCC scoping, and improved CRD validation.

    The release supports installation with kubectl or Helm. The official spoc CLI is available for amd64, arm64, ppc64le, and s390x.

    All release artifacts are signed and include SLSA build provenance, with verification available for container images, binaries, the SBOM, and Helm chart.

    Security Profiles Operator v1.1.0 release

  • IBM Cloud Internet Service Webhook for Cert Manager / Updated image

    If you are using Cert Manager with IBM Cloud Internet Service Webhook for Cert Manager link, please update to use quay.io/ibm/cert-manager-webhook-ibmcis:0.2.4

  • 2026-SEPT: Additions IBM Power Open Source Images on the IBM Container Registry

    The IBM Linux on Power team has released some new open source container images into the IBM Container Registry (ICR). These images are available for no-charge and can be used in your development and production environments.

    Image NameTag NameProject LicensesImage Pull CommandLast Published On
    pgvector0.8.6PostgreSQLpodman pull icr.io/ppc64le-oss/pgvector-ppc64le:0.8.62026-09-03
    pulsarv4.2.3Apache-2.0podman pull icr.io/ppc64le-oss/pulsar-ppc64le:4.2.32026-09-01
    opensearch-dashboards3.6.0Apache-2.0podman pull icr.io/ppc64le-oss/opensearch-dashboards-ppc64le:3.6.02026-08-24
    openrag-opensearch0.6.0Apache-2.0podman pull icr.io/ppc64le-oss/openrag-opensearch-ppc64le:0.6.02026-08-13
    openrag-backend0.6.0Apache-2.0podman pull icr.io/ppc64le-oss/openrag-backend-ppc64le:0.6.02026-08-12
    openrag-frontend0.6.0Apache-2.0podman pull icr.io/ppc64le-oss/openrag-frontend-ppc64le:0.6.02026-08-12
    litellm1.92.0MITpodman pull icr.io/ppc64le-oss/litellm-ppc64le :1.92.02026-07-28
    hapi-fhir8.10.0Apache-2.0podman pull icr.io/ppc64le-oss/hapi-fhir-ppc64le:8.10.02026-07-21
    filebeatv9.4.5Apache License 2.0podman pull icr.io/ppc64le-oss/filebeat-ppc64le:9.4.52026-09-17
    logstashv9.4.5Apache License 2.0podman pull icr.io/ppc64le-oss/logstash-ppc64le:9.4.52026-09-17

    Refer to https://community.ibm.com/community/user/blogs/priya-seth/2023/04/05/open-source-containers-for-power-in-icr for more details.

    If you need opensource software enabled on IBM Power, reach out at https://www.ibm.com/power/resources/isv/enablement-request/

  • Official IBM Power (ppc64le) Support lands for Vault, Vault CSI Provider and Vault Secrets Operator on Kubernetes and OpenShift

    The IBM Power and Vault team have delivered official ppc64le (IBM Power Linux) support for the Vault Kubernetes ecosystem, expanding platform coverage to all four major architectures: amd64, arm64, ppc64le, and s390x.

    New multi-architecture UBI images are now available in both IBM Container Registry (ICR) and Red Hat Quay for:

    • vault-secrets-operator 1.5.1
    • vault-csi-provider 1.7.4
    • vault-k8s 1.7.6

    This is an important milestone for customers running Kubernetes workloads on IBM Power, enabling deployment of supported Vault integrations without custom builds and helping standardize secrets management across heterogeneous infrastructure.

    References

    Vault Secrets Operator 1.5.1 release: https://github.com/hashicorp/vault-secrets-operator/releases/tag/v1.5.1 

    Vault Helm releases: https://github.com/hashicorp/vault-helm/releases

  • IBM Power and Red Hat OpenShift’s multi-architecture capabilities

    Sumeet’s blog dives into IBM Power and Red Hat OpenShift’s multi-architecture capabilities, teams can run Power and x86 workloads within a common operational framework, reducing platform silos while accelerating application modernization that fits your needs.

    Reference https://community.ibm.com/community/user/blogs/balani-sumeet/2026/08/24/hybrid-cloud-multi-arch-operations-with-ibm-power

  • Helm v4 Location to Download

    If you need Helm v4 from Red Hat, you can grab the package manager for Kubernetes at https://mirror.openshift.com/pub/cgw/helm/4.1.4/

  • Using an EgressIP with UserDefinedNetwork on OpenShift Container Platform for IBM Power

    As OpenShift Container Platform (OCP) adoption continues to grow on IBM Power systems, customers are increasingly looking to combine advanced networking capabilities with on-premises PowerVM deployments. One area that often generates questions is how EgressIP behaves with a primary UserDefinedNetwork (UDN) and how traffic is routed from workloads attached to that network.

    This walkthrough demonstrates how to deploy a Layer 2 primary UserDefinedNetwork, assign an EgressIP, and validate that outbound traffic from a pod traverses the UDN interface rather than the cluster default network. The example is particularly relevant for OpenShift on IBM Power, including PowerVM environments managed through PowerVC.

    Why UserDefinedNetworks Matter

    UserDefinedNetworks (UDNs) allow OpenShift administrators to create custom pod networking domains separate from the cluster’s default network. When deployed as a Primary network, the UDN becomes the pod’s default network interface and routing domain.

    This can be useful for:

    • Application isolation
    • Dedicated routing paths
    • Network segmentation
    • Multi-tenant environments
    • Advanced EgressIP scenarios

    For organizations running OCP on IBM Power, UDNs provide the same flexibility available on other supported OpenShift platforms while leveraging the scale and resiliency of Power infrastructure.

    Prerequisites

    Before creating the EgressIP resource, choose a valid IP address that:

    • Is currently unused
    • Is routable from your worker nodes
    • Exists on the same Layer 2 segment as the node interface hosting the EgressIP

    ⚠️ Replace EGRESS_IP_ADDRESS with a valid address from your environment.

    For on-premises PowerVM deployments, this IP should belong to the same network segment as the worker nodes that will advertise the EgressIP.

    Create the UDN-Enabled Namespace

    Namespaces that use a primary UDN must be labeled appropriately.

    cat <<'EOF' | oc apply -f -
    apiVersion: v1
    kind: Namespace
    metadata:
      name: egressip-udn-repro
      labels:
        k8s.ovn.org/primary-user-defined-network: ""
        egressip-test: "true"
    EOF

    The label k8s.ovn.org/primary-user-defined-network: “” signals that workloads in the namespace should attach to a primary UserDefinedNetwork.

    The additional label egressip-test: “true” is later used by the EgressIP namespaceSelector.

    Create a Layer 2 Primary UserDefinedNetwork

    Create a primary Layer 2 UDN:

    cat <<'EOF' | oc apply -f -
    apiVersion: k8s.ovn.org/v1
    kind: UserDefinedNetwork
    metadata:
      name: repro-udn-l2
      namespace: egressip-udn-repro
    spec:
      topology: Layer2
      layer2:
        role: Primary
        subnets:
          - "10.100.0.0/24"
    EOF

    Wait for the network to become ready:

    oc wait userdefinednetwork repro-udn-l2 -n egressip-udn-repro \
      --for condition=NetworkReady=True --timeout=60s

    💡 Ensure the UDN subnet does not overlap with the cluster pod network, service network, or OVN join subnets. In this example, 10.100.0.0/24 is used as the UDN subnet.

    Enable EgressIP Assignment on a Worker Node

    Label a worker node so OVN-Kubernetes can host EgressIP assignments:

    oc label node worker-0 k8s.ovn.org/egress-assignable=””

    You can verify the label using:

    oc get nodes –show-labels

    Create the EgressIP Resource

    cat <<'EOF' | oc apply -f -
    apiVersion: k8s.ovn.org/v1
    kind: EgressIP
    metadata:
      name: repro-egressip
    spec:
      egressIPs:
        - "EGRESS_IP_ADDRESS"
      namespaceSelector:
        matchLabels:
          egressip-test: "true"
    EOF

    In a lab environment, an unused address was selected from the worker node network and associated with the node through an additional interface configured in PowerVC – and EGRESS_IP_ADDRESS was updated to match the unused address that was routable on the network.

    Verify EgressIP Assignment

    Confirm that OVN successfully assigned the EgressIP:

    oc get egressip repro-egressip -o yaml

    Review the status section and verify the address is assigned to the intended worker node.

    Example:

    status:
           items:
           – egressIP: 192.168.1.50
             node: worker-0

    Deploy a Test Pod

    Create a simple UBI-based pod:

    cat <<'EOF' | oc apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      name: egress-test-pod
      namespace: egressip-udn-repro
      labels:
        app: egress-test
    spec:
      containers:
        - name: curl
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command: ["sleep", "infinity"]
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
    EOF

    Wait until the pod becomes ready:

    oc wait pod egress-test-pod -n egressip-udn-repro --for condition=Ready --timeout=120s

    Install curl

    Access the pod:

    oc rsh -n egressip-udn-repro egress-test-pod

    Install curl:

    microdnf –setopt=cachedir=/tmp/dnf-cache –setopt=keepcache=0 install -y curl

    Validate External Connectivity

    Generate outbound traffic:

    curl -sv https://google.com/ –connect-timeout 10

    You can replace the endpoint with any reachable external service that allows source IP verification.

    Inspect Routing Inside the Pod

    Review the routing table:

    cat /proc/net/route

    Example output:

    Iface      Destination Gateway
         ovn-udn1   00000000    0100640A
         ovn-udn1   0000640A    00000000
         eth0       0000800A    0102800A

    Decoded, the routes appear as:

    DestinationGatewayInterface
    0.0.0.010.100.0.1ovn-udn1
    10.100.0.0/24Directovn-udn1
    10.128.0.0/2210.128.2.1eth0
    100.64.0.0/1610.128.2.1eth0
    100.65.0.0/1610.100.0.1ovn-udn1
    172.30.0.0/1610.100.0.1ovn-udn1

    The key observation is that the default route points to ovn-udn1, confirming that the primary UDN is acting as the pod’s primary network.

    Verify Traffic Flow

    Examine interface statistics:

    cat /proc/net/dev

    Example:

         Inter-| Receive | Transmit
         …
         eth0:      446      5
         ovn-udn1: 21828173 4698

    The significantly higher packet and byte counts on ovn-udn1 indicate that application traffic is flowing through the UserDefinedNetwork rather than the cluster default interface.

    This validates that:

    1. The pod is attached to the primary UDN.
    2. The default route is installed through ovn-udn1.
    3. Outbound traffic follows the UDN path.
    4. EgressIP can be applied to workloads selected through the namespace selector.

    Conclusion

    OpenShift’s UserDefinedNetwork capability provides a powerful way to create dedicated networking domains for applications while still taking advantage of platform services such as EgressIP. On IBM Power deployments, including PowerVM-based environments, this enables architects to combine network isolation with consistent outbound IP presentation.

    In this validation, a primary Layer 2 UDN successfully became the pod’s default network, routes were installed through ovn-udn1, and traffic counters confirmed that outbound connections traversed the UDN. Combined with EgressIP assignment, this offers a flexible pattern for workloads that require network segmentation and predictable egress behavior on OpenShift running on IBM Power.