This post helps those who are stuck with “Application is not available” on the OpenShift Console on IBM Virtual Private Cloud (VPC).

First, when you access the OpenShift Console you’ll see https://console-openshift-console.hidden.eu-gb.containers.appdomain.cloud/dashboards

Steps

1. Find your worker nodes

$ oc get nodes -l node-role.kubernetes.io/worker
NAME         STATUS   ROLES           AGE   VERSION
worker0   Ready    master,worker   28h   v1.23.5+3afdacb
worker1   Ready    master,worker   28h   v1.23.5+3afdacb

2. Launch a debug pod to the node/worker0 and execute a chroot, and curl to confirm it times out.

$ oc debug node/worker0                                                               
Starting pod/1024204-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.242.0.4
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
curl google.com -v -k
* About to connect() to google.com port 80 (#0)
*   Trying 216.58.212.238...

If the curl command never completes, then you probably don’t have the VPC set for egress.

3. Navigate to https://cloud.ibm.com/vpc-ext/network/subnet/

4. Find your subnet, click on Public Gateway

5. Retry accessing your Console (You can also retry from the command line oc debug). You should now see the dashboard (note it may need to retry the CrashBackOffLoop for the pod, so it may be a few minutes).

Appendix: Checking your Console URL

If you don’t know your external console URL, you can retrieve it from oc.

$ oc -n openshift-config-managed get cm console-public -o jsonpath='{.data.consoleURL}'
https://console-openshift-console.hidden.eu-gb.containers.appdomain.cloud

Appendix: Checking Access Tokens

If you are using OauthAccessTokens in your environment, and you closed your display, you can always get a view (as a kubeadmin) of the current access tokens using the OpenShift command line.

$ oc get oauthaccesstokens -A              
NAME                                                 USER NAME                        CLIENT NAME                CREATED   EXPIRES                         REDIRECT URI                                                              SCOPES
sha256~-m   IAM#yyy@ibm.com    openshift-browser-client   12m       2022-06-22 15:00:38 +0000 UTC   https://hiddene.eu-gb.containers.cloud.ibm.com:31871/oauth/token/display   user:full
sha256~x   IAM#g@ibm.com    openshift-browser-client   10m       2022-06-22 15:02:24 +0000 UTC   https://hiddene.eu-gb.containers.cloud.ibm.com:31871/oauth/token/display   user:full
sha256~z   IAM#x@us.ibm.com          openshift-browser-client   171m      2022-06-22 12:21:30 +0000 UTC   https://hiddene.eu-gb.containers.cloud.ibm.com:31871/oauth/token/display   user:full
sha256~z   IAM#y@ibm.com   openshift-browser-client   131m      2022-06-22 13:01:18 +0000 UTC   https://hiddene.eu-gb.containers.cloud.ibm.com:31871/oauth/token/display   user:full
sha256~y   IAM#y@ibm.com   openshift-browser-client   84m       2022-06-22 13:48:29 +0000 UTC   https://hiddene.eu-gb.containers.cloud.ibm.com:31871/oauth/token/display   user:full
sha256~x   IAM#y@ibm.com   openshift-browser-client   130m      2022-06-22 13:02:25 +0000 UTC   https://hiddene.eu-gb.containers.cloud.ibm.com:31871/oauth/token/display   user:full

Appendix: Checking the OAuth Well Known

To check the well known oauth endpoints, check https://hidden-e.eu-gb.containers.cloud.ibm.com:30603/.well-known/oauth-authorization-server

In OpenShift, the kube-scheduler binds a unit of work (Pod) to a Node. The scheduler reads from a scheduling queue the work, retrieves the current state of the cluster, scores the work based on the scheduling rules (from the policy) and the cluster’s state, and prioritizes binding the Pod to a Node.

These nodes are scheduled based on an instantaneous read of the policy and the environment and a best-estimation placement of the Pod on a Node. With best estimate at the time, these clusters are constantly changing shape and context; there is a need to deschedule and schedule the Pod anew. 

There are four actors in the Descheduler:

1. User configures the KubeDescheduler resource
2. Operator creates the Descheduler Deployment
3. Descheduler run on a set interval and re-evaluates the scheduled Pod and Node and Policy, setting an eviction if the Pod should be removed based on the Descheduler Policy.
4. Pod is removed (unbound).

Thankfully, OpenShift has a Descheduler Operator that more easily facilitates the unbinding of a Pod from a Node based on a cluster-wide configuration of the KubeDescheduler CustomResource. In a single cluster, there is at most one configured KubeDescheduler named cluster (it has to be fixed), and configures one or more Descheduler Profiles. 

Descheduler Profiles are predefined and available in the profiles folder – DeschedulerProfile:

AffinityAndTaints	Balance pods based on node taint violations
TopologyAndDuplicates	Spreads pods evenly among nodes based on topology constraints and duplicate replicates on the same node   The profile cannot be used with SoftTopologyAndDuplicates.
SoftTopologyAndDuplicates	Spreads pods with prior with soft constraints   The profile cannot be used with TopologyAndDuplicates.
LifecycleAndUtilization	Balances pods based on node resource usage   This profile cannot be used with DevPreviewLongLifecycle
EvictPodsWithLocalStorage	Enables pods with local storage to be evicted by the descheduler by all other profiles
EvictPodsWithPVC	Prevents pods with PVCs from being evicted by all other profiles
DevPreviewLongLifecycle	Lifecycle management for pods that are ‘long running’   This profile cannot be used with LifecycleAndUtilization

There must be one or more DeschedulerProfile specified, and there cannot be any duplicates entries. There are two possible mode values – Automatic and Predictive. You have to go the Pod to check the output to see what is Predicted or is Completed.

The DeschedulerOperator excludes the openshift-*, kube-system and hypershift namespaces.

Steps

1.   Login to your OpenShift Cluster 

oc login --token=sha256~1111-g --server=https://api..sslip.io:6443

2. Create a Pod that indicates it’s available for eviction using the annotation descheduler.alpha.kubernetes.io/evict: “true” and is updated for the proper node name.

cat << EOF > pod.yaml 
kind: Pod
apiVersion: v1
metadata:
  annotations:
    descheduler.alpha.kubernetes.io/evict: "true"
  name: demopod1
  labels:
    foo: bar
spec:
  containers:
  - name: pause
    image: docker.io/ibmcom/pause-ppc64le:3.1
EOF
oc apply -f pod.yaml 
pod/demopod1 created

• 3. Create the KubeDescheduler CR with a Descheduling Interval of 60 seconds and Pod Lifetime of 1m.

cat << EOF > kd.yaml 
apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
  name: cluster
  namespace: openshift-kube-descheduler-operator
spec:
  logLevel: Normal
  mode: Predictive
  operatorLogLevel: Normal
  deschedulingIntervalSeconds: 60
  profileCustomizations:
    podLifetime: 1m0s
  observedConfig:
    servingInfo:
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      minTLSVersion: VersionTLS12
  profiles:
    - LifecycleAndUtilization
  managementState: Managed
EOF
oc apply -f kd.yaml

• 4. Get the Pods in the openshift-kube-descheduler-operator

oc get pods -n openshift-kube-descheduler-operator                              
NAME                                    READY   STATUS    RESTARTS   AGE
descheduler-f479c5669-5ffxl             1/1     Running   0          2m7s
descheduler-operator-85fc6666cb-5dfr7   1/1     Running   0          27h

• 5. Check the Logs for the descheduler pod

oc -n openshift-kube-descheduler-operator logs descheduler-f479c5669-5ffxl
I0506 19:59:10.298440       1 pod_lifetime.go:110] "Evicted pod because it exceeded its lifetime" pod="minio-operator/console-7bc65f7dd9-q57lr" maxPodLifeTime=60
I0506 19:59:10.298500       1 evictions.go:158] "Evicted pod in dry run mode" pod="default/demopod1" reason="PodLifeTime"
I0506 19:59:10.298532       1 pod_lifetime.go:110] "Evicted pod because it exceeded its lifetime" pod="default/demopod1" maxPodLifeTime=60
I0506 19:59:10.298598       1 toomanyrestarts.go:90] "Processing node" node="master-0.rdr-rhop-.sslip.io"
I0506 19:59:10.299118       1 toomanyrestarts.go:90] "Processing node" node="master-1.rdr-rhop.sslip.io"
I0506 19:59:10.299575       1 toomanyrestarts.go:90] "Processing node" node="master-2.rdr-rhop.sslip.io"
I0506 19:59:10.300385       1 toomanyrestarts.go:90] "Processing node" node="worker-0.rdr-rhop.sslip.io"
I0506 19:59:10.300701       1 toomanyrestarts.go:90] "Processing node" node="worker-1.rdr-rhop.sslip.io"
I0506 19:59:10.301097       1 descheduler.go:287] "Number of evicted pods" totalEvicted=5

This article shows a simple case for the Descheduler and you can see how it ran a dry run and showed it would evict five pods.

A brief Operator training I gave to my team resulted in these notes. Thanks to many others in the reference section.

An Operator codifies the tasks commonly associated with administrating, operating, and supporting an application.  The codified tasks are event-driven responses to changes (create-update-delete-time) in the declared state relative to the actual state of an application, using domain knowledge to reconcile the state and report on the status.

Operators are used to execute basic and advanced operations:

Basic (Helm, Go, Ansible)

1. Installation and Configuration
2. Uninstall and Destroy
3. Seamless Upgrades

Advanced (Go, Ansible)

1. Application Lifecycle (Backup, Failure Recovery)
2. Monitoring, Metrics, Alerts, Log Processing, Workload Analysis
3. Auto-scaling: Horizontal and Vertical
4. Event (Anomaly) Detection and Response (Remediation)
5. Scheduling and Tuning
6. Application Specific Management
7. Continuous Testing and Chaos Monkey

Helm operators wrap helm charts in a simplistic view of the operation pass-through helm verbs, so one can install, uninstall, destroy, and upgrade using an Operator.

There are four actors in the Operator Pattern.

1. Initiator – The user who creates the Custom Resource
2. Operator – The Controller that operates on the Operand
3. Operand – The target application
4. OpenShift and Kubernetes Environment

Each Operator operates on an Operand using Managed Resources (Kubernetes and OpenShift) to reconcile states.  The states are described in a domain specific language (DSL) encapsulated in a Custom Resource to describe the state of the application:

1. spec – The User communicates to the Operator the desired state (Operator reads)
2. status – The Operator communicates back to the User (Operator writes)

$ oc get authentications cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Authentication
metadata:
  annotations:
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
spec:
  oauthMetadata:
    name: ""
  serviceAccountIssuer: ""
  type: ""
  webhookTokenAuthenticator:
    kubeConfig:
      name: webhook-authentication-integrated-oauth
status:
  integratedOAuthMetadata:
    name: oauth-openshift

While not limited to writing spec and status, if we think spec is initiator specified, and if we think status is operator written, then we limit the chances of creating an unintended reconciliation loop.

The DSL is specified as Custom Resource Definition:

$ oc get crd machinehealthchecks.machine.openshift.io -o=yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
spec:
  conversion:
    strategy: None
  group: machine.openshift.io
  names:
    kind: MachineHealthCheck
    listKind: MachineHealthCheckList
    plural: machinehealthchecks
    shortNames:
    - mhc
    - mhcs
    singular: machinehealthcheck
  scope: Namespaced
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: 'MachineHealthCheck'
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource'
            type: string
          metadata:
            type: object
          spec:
            description: Specification of machine health check policy
            properties:
              expectedMachines:
                description: total number of machines counted by this machine health
                  check
                minimum: 0
                type: integer
              unhealthyConditions:
                description: UnhealthyConditions contains a list of the conditions.
                items:
                  description: UnhealthyCondition represents a Node.
                  properties:
                    status:
                      minLength: 1
                      type: string
                    timeout:
                      description: Expects an unsigned duration string of decimal
                        numbers each with optional fraction and a unit suffix, eg
                        "300ms", "1.5h" or "2h45m". Valid time units are "ns", "us"
                        (or "µs"), "ms", "s", "m", "h".
                      pattern: ^([0-9]+(\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$
                      type: string
                    type:
                      minLength: 1
                      type: string
                  type: object
                minItems: 1
                type: array
            type: object

For example, these operators manage the applications by orchestrating operations based on changes to the CustomResource (DSL):

Operator Type/Language	What it does	Operations
cluster-etcd-operator go	Manages etcd in OpenShift	Install Monitor Manage
prometheus-operator go	Manages Prometheus monitoring on a Kubernetes cluster	Install Monitor Manage Configure
cluster-authentication-operator go	Manages OpenShift Authentication	Manage Observe

As a developer, we’re going to follow a common development pattern:

1. Implement the Operator Logic (Reconcile the operational state)
2. Bake Container Image
3. Create or regenerate Custom Resource Definition (CRD)
4. Create or regenerate Role-based Access Control (RBAC)
   1. Role
   1. RoleBinding
5. Apply Operator YAML

Note, we’re not necessarily writing business logic, rather operational logic.

There are some best practices we follow:

1. Develop one operator per application
   1. One CRD per Controller. Created and Fit for Purpose. Less Contention.
   1. No Cross Dependencies.
2. Use Kubernetes Primitives when Possible
3. Be Backwards Compatible
4. Compartmentalize features via multiple controllers
   1. Scale = one controller
   1. Backup = one controller
5. Use asynchronous metaphors with the synchronous reconciliation loop
   1. Error, then immediate return, backoff and check later
   1. Use concurrency to split the processing / state
6. Prune Kubernetes Resources when not used
7. Apps Run when Operators are stopped
8. Document what the operator does and how it does it
9. Install in a single command

We use the Operator SDK – one it’s supported by Red Hat and the CNCF.

operator-sdk: Which one? Ansible and Go

Kubernetes is authored in the Go language. Currently, OpenShift uses Go 1.17 and most operators are implemented in Go. The community has built many go-based operators, we have much more support on StackOverflow and a forum.

	Ansible	Go
Kubernetes Support	Cached Clients	Solid, Complete and Rich Kubernetes Client
Language Type	Declarative – describe the end state	Imperative – describe how to get to the end state
Operator Type	Indirect Wrapped in the Ansible-Operator	Direct
Style	Systems Administration	Systems Programming
Performance	Link	~4M at startup Single layer scratch image
Security	Expanded Surface Area	Limited Surface Area

Go is ideal for concurrency, strong memory management, everything is baked into the executable deliverable – it’s in memory and ready-to-go. There are lots of alternatives to code NodeJS, Rust, Java, C#, Python. The OpenShift Operators are not necessarily built on the Operator SDK.

Summary

We’ve run through a lot of detail on Operators and learned why we should go with Go operators.

Reference

1. CNCF Operator White Paper https://github.com/cncf/tag-app-delivery/blob/main/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md
2. Operator pattern https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
3. Operator SDK Framework https://sdk.operatorframework.io/docs/overview/
4. Kubernetes Operators 101, Part 2: How operators work https://developers.redhat.com/articles/2021/06/22/kubernetes-operators-101-part-2-how-operators-work?source=sso#
5. Build Kubernetes with the Right Tool https://cloud.redhat.com/blog/build-your-kubernetes-operator-with-the-right-tool https://hazelcast.com/blog/build-your-kubernetes-operator-with-the-right-tool/
6. Build Your Kubernetes Operator with the Right Tool
7. Operator SDK Best Practices https://sdk.operatorframework.io/docs/best-practices/
8. Google Best practices for building Kubernetes Operators and stateful apps https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-building-kubernetes-operators-and-stateful-apps
9. Kubernetes Operator Patterns and Best Practises https://github.com/IBM/operator-sample-go
10. Fast vs Easy: Benchmarking Ansible Operators for Kubernetes https://www.ansible.com/blog/fast-vs-easy-benchmarking-ansible-operators-for-kubernetes
11. Debugging a Kubernetes Operator https://www.youtube.com/watch?v=8hlx6F4wLAA&t=21s
12. Contributing to the Image Registry Operator https://github.com/openshift/cluster-image-registry-operator/blob/master/CONTRIBUTING.md
13. Leszko’s OperatorCon Presentation
    1. YouTube https://www.youtube.com/watch?v=hTapESrAmLc
    1. GitHub Repo for Session: https://github.com/leszko/build-your-operator