As many know, Go is a designed for performance with an emphasis on memory management and garbage collection. When used within cgroups with Kubernetes and Red Hat OpenShift Go maximizes for the available memory on the node and the available processors. This approach, as noted by Uber’s automaxprocs, a shared system can see slightly degraded performance when allocated CPUs are not limited to the actually available CPUs (e.g., a prescribed limit).

Using environment variables, Go lets a user control Memory limits and processor limits.

GOMEMLIMIT limits the Go heap and all other runtime memory runtime/debug.SetMemoryLimit

GOMAXPROCS limits the number of operating system threads that can execute user-level Go code simultaneously.

There is an opensource go packages to control GOMAXPROCS automatically when used with cgroups called automaxproces.

In OpenShift/Kubernetes, there is a concept of spec.containers[].resources.limits for cpus and memory, as described in the article Resource Management for Pods and Containers.

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: myimage
    resources:
      limits:
        memory: "128Mi"
        cpu: 2

To facilitate sharing these details to a container Kubernetes provides the downwardAPI. The downwardAPI provides the details as an environment variableor a file.

To see how this works in combination:

1. Create a yaml test.yaml with resources.limits and env.valueFrom.fieldRef.fieldPath set to the GOMEMLIMIT and GOMAXPROCS value you want.

kind: Namespace
apiVersion: v1
metadata:
  name: demo
---
apiVersion: v1
kind: Pod
metadata:
  name: dapi-go-limits
  namespace: demo
spec:
  containers:
    - name: test-container
      image: registry.access.redhat.com/ubi8/pause
      resources:
        limits:
          memory: 128Mi
          cpu: "2"
      command:
        - sh
        - '-c'
      args:
        - >-
          while true; do echo -en '\n'; printenv GOMEMLIMIT; printenv GOMAXPROCS
          sleep 10; done;
      env:
        - name: GOMEMLIMIT
          valueFrom:
            resourceFieldRef:
              containerName: test-container
              resource: limits.memory
        - name: GOMAXPROCS
          valueFrom:
            resourceFieldRef:
              containerName: test-container
              resource: limits.cpu
  restartPolicy: Never

2. Apply the file to the oc apply -f test.yaml

3. Check the logs file

$ oc -n demo logs pod/dapi-go-limits
134217728
2

4. Delete the pod when you are done with the demonstration

$ oc -n demo delete pod/dapi-go-limits
pod "dapi-go-limits" deleted

There is a clear / easy way to control go runtime configuration.

Reference

• https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/
• https://stackoverflow.com/questions/17853831/what-is-the-gomaxprocs-default-value
• https://github.com/uber-go/automaxprocs#performance
• https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/

Here are my notes for setting up the SIG’s nfs-provisioner. You should follow these directions to setup the nfs-provisioner kubernetes-sigs/nfs-subdir-external-provisioner.

1. Clone the nfs-subdir-external-provisioner

git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git

2. If you haven’t already, you may need to create the nfs-provisioner namespace.

a. Create the ns.yaml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: nfs-provisioner
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/enforce-version: v1.24
  name: nfs-provisioner

b. create the namespace

oc apply -f ns.yaml

c. annotate the namespace

oc label namespace/nfs-provisioner security.openshift.io/scc.podSecurityLabelSync=false --overwrite=true
oc label namespace/nfs-provisioner pod-security.kubernetes.io/enforce=privileged --overwrite=true
oc label namespace/nfs-provisioner pod-security.kubernetes.io/audit=privileged --overwrite=true
oc label namespace/nfs-provisioner pod-security.kubernetes.io/warn=privileged --overwrite=true

3. Change to the deploy/ directory

cd nfs-subdir-external-provisioner/deploy

4. Update the namespace default to nfs-provisioner for deployment.yaml

5. On the Bastion server, look at ocp4-helpernode/helpernode_vars.yaml for the helper.ipaddr value.

helper:
  networkifacename: env3
  name: "bastion-0"
  ipaddr: "193.168.200.15"

6. Update the deployment with the NFS_SERVER using the helper.ipaddr and the NFS_PATH /export. It should look like the following:

    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: k8s.gcr.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 193.168.200.15
            - name: NFS_PATH
              value: /export
      volumes:
        - name: nfs-client-root
          nfs:
            server: 193.168.200.15
            path: /export

v4.0.2 supports ppc64le.

Be sure to remove the namespace: default

7. Create the deployment

oc apply -f deployment.yaml
deployment.apps/nfs-client-provisioner created

8. Get the pods

oc get pods
NAME                                     READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-b8764c6bb-mjnq9   1/1     Running   0          36s

9. Setup Authorization

NAMESPACE=`oc project -q`
sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./rbac.yaml 
oc create -f rbac.yaml
oc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:$NAMESPACE:nfs-client-provisioner

10. Create the storage class file

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}" # waits for nfs.io/storage-path annotation, if not specified will accept as empty string.
  onDelete: delete

11. Apply the StorageClass

oc apply -f sc.yml

12. Then you can deploy the PV and PVC files/6_EvictPodsWithPVC_dp.yml

References

• git repo

Per the IBM Cloud Kubernetes cluster-api provider, I followed the raw instructions with some amendments.

Steps

1. Provision an Ubuntu 20.04 image.

2. Update the apt repository

$ apt update

3. Install the dependencies (more than what’s in the instructions)

$ apt install qemu-kvm libvirt-daemon-system libvirt-clients virtinst cpu-checker libguestfs-tools libosinfo-bin make git unzip ansible python3-pip

4. Clone the image-builder repo

$ git clone https://github.com/kubernetes-sigs/image-builder.git

5. Change to the capi image

$ cd image-builder/images/capi

6. Make the deps-raw to confirm everything is working.

$ make deps-raw

7. Create the ubuntu-2004 image.

$ make build-qemu-ubuntu-2004

Once complete you’ll see:

==> qemu: Running post-processor: custom-post-processor (type shell-local)
==> qemu (shell-local): Running local shell script: /tmp/packer-shell078717884
Build 'qemu' finished after 12 minutes 8 seconds.

==> Wait completed after 12 minutes 8 seconds

==> Builds finished. The artifacts of successful builds are:
--> qemu: VM files in directory: ./output/ubuntu-2004-kube-v1.22.9
--> qemu: VM files in directory: ./output/ubuntu-2004-kube-v1.22.9

8. Append the .qcow2 extension

$ mv ./output/ubuntu-2004-kube-v1.22.9/ubuntu-2004-kube-v1.22.9 ./output/ubuntu-2004-kube-v1.22.9/ubuntu-2004-kube-v1.22.9.qcow2

You can now upload the output to IBM Cloud Object Storage.

A couple quick tips:

• If you see any warnings, you can get advanced details using export PACKER_LOG=1 which puts out the full packer logging. see Packer
• KVM module not found indicates you are running in a nested KVM, you’ll have to swap out of the VM and enable nested KVM. Fedora: Docs
• Adding a VM to VPC is documented here Console: customImage