Paul Bastide

Analyzing Memory Hacks with Kubernetes and OpenShift and Linux

Written by

Paul

in

I’ve had to run a number of queries for an OpenShift Cluster, Kubernetes and Linux recently, and here are my helpful queries:

Node Memory

If you want to check the Memory on each Node in your OpenShift Cluster, you can run the following oc command:

$ oc get nodes -o json | jq -r '.items[] | "\(.metadata.name) - \(.status.capacity.memory)"'
master-0.ocp-power.xyz - 16652928Ki
master-1.ocp-power.xyz - 16652928Ki
master-2.ocp-power.xyz - 16652928Ki
worker-0.ocp-power.xyz - 16652928Ki
worker-1.ocp-power.xyz - 16652928Ki

Node Memory Pressure

If you want to check the Memory usage on each Node in your OpenShift Cluster, you can run the following oc command: 

Memory Pressure Per node:
$ oc adm top node --show-capacity=true
NAME  CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%  
master-0.ocp-power.xyz    1894m 15272Mi 93%
master-1.ocp-power.xyz    1037m 8926Mi  54%
master-2.ocp-power.xyz    1563m 10953Mi 67%
worker-0.ocp-power.xyz    1523m 6781Mi  41%
worker-1.ocp-power.xyz    933m  6746Mi  41%

Top Memory Usage per Pods

If you want to check the Top Memory Usage per Pod, you can run the following command:

$ oc adm top pod -A --sort-by='memory'
 NAMESPACE  NAME CPU(cores)   MEMORY(bytes)  
15% - openshift-kube-apiserver   kube-apiserver-master-2.ocp-power.xyz   386m 2452Mi 
11% - openshift-kube-apiserver   kube-apiserver-master-0.ocp-power.xyz   225m 1924Mi 
10% - openshift-kube-apiserver   kube-apiserver-master-1.ocp-power.xyz   239m 1720Mi

List Container Memory Details per Pod

If you want to see the breakdown of Memory usage, you can use the following kubectl command:

$ kubectl top pod kube-apiserver-master-2.ocp-power.xyz -n openshift-kube-apiserver --containers
POD  NAME  CPU(cores)   MEMORY(bytes)  
kube-apiserver-master-2.ocp-power.xyz   POD   0m   0Mi
kube-apiserver-master-2.ocp-power.xyz   kube-apiserver514m 2232Mi 
kube-apiserver-master-2.ocp-power.xyz   kube-apiserver-cert-regeneration-controller   25m  51Mi   
kube-apiserver-master-2.ocp-power.xyz   kube-apiserver-cert-syncer0m   28Mi   
kube-apiserver-master-2.ocp-power.xyz   kube-apiserver-check-endpoints7m   41Mi   
kube-apiserver-master-2.ocp-power.xyz   kube-apiserver-insecure-readyz0m   16Mi

Checking the High and Low Memory Limits on a Linux Host

If you want to check the memory usage on a host in Gigabytes (including the max allocation), you can run the free command:

$ free -g -h -l
              total        used        free      shared  buff/cache   available
Mem:           15Gi        10Gi       348Mi       165Mi       5.2Gi       4.8Gi
Low:           15Gi        15Gi       348Mi
High:            0B          0B          0B
Swap:            0B          0B          0B

Use Observe > Metrics

If you have Metrics enabled, login to your OpenShift Dashboard, and click on Observe > Metrics and use one of the following

sum(node_memory_MemAvailable_bytes) by (instance) / 1024 / 1024 / 1024
:node_memory_MemAvailable_bytes:sum

Check Memory Usage on the CoreOS Nodes

If you want to check the memory details on each CoreOS node, you can use the following hack to SSH in and output the details.

$ for HN in $(oc get nodes -o json | jq -r '.items[].status.addresses[] | select(.type=="Hostname").address')
do
   echo HOSTNAME: $HN
   ssh core@$HN 'cat /proc/meminfo'
done

HOSTNAME: master-0.ocp-power-xyz
MemTotal:       16652928 kB
MemFree:          265472 kB
MemAvailable:     933248 kB
Buffers:             384 kB
Cached:          1387584 kB
SwapCached:            0 kB
Active:           688000 kB
Inactive:        7832192 kB
Active(anon):     120448 kB
Inactive(anon):  7307392 kB

Check Top in Batch on the CoreOS Nodes

If you want to check the Memory using Top (batch) on each CoreOS node, you can use the following hack to SSH in and output the details: (refer to link)

for HN in $(oc get nodes -o json | jq -r '.items[].status.addresses[] | select(.type=="Hostname").address')
do
   echo
   echo HOSTNAME: $HN
   ssh core@$HN 'top -b -d 5 -n 1 -E g -o +%MEM'
   sleep 10
done

HOSTNAME: master-0.ocp-power.xyz
top - 23:41:40 up 7 days, 11:58,  0 users,  load average: 1.60, 2.24, 2.74
Tasks: 390 total,   1 running, 389 sleeping,   0 stopped,   0 zombie
%Cpu(s): 48.1 us, 10.6 sy,  0.0 ni, 39.4 id,  1.2 wa,  0.0 hi,  0.6 si,  0.0 st
GiB Mem :     15.9 total,      0.2 free,     14.3 used,      1.3 buff/cache
GiB Swap:      0.0 total,      0.0 free,      0.0 used.      0.8 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1018800 root      20   0 2661824   1.7g  21696 S   0.0  10.5   2896:40 kube-ap+
  42247 root      20   0   11.3g   1.1g  14272 S  16.7   7.1   1483:55 etcd
   1704 root      20   0 3984384 220480  31680 S  11.1   1.3   3289:24 kubelet

Pod Metrics (Thanks StackOverflow)

If you want to get raw cpu and memory metrics from OpenShift, you can run the following: 

kubectl -n openshift-etcd get --raw /apis/metrics.k8s.io/v1beta1/namespaces/openshift-etcd/pods/etcd-master-0.ocp-power.xyz | jq
{
  "kind": "PodMetrics",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "name": "etcd-master-0.ocp-power.xyz",
    "namespace": "openshift-etcd",
    "creationTimestamp": "2022-06-25T00:10:58Z",
    "labels": {
      "app": "etcd",
      "etcd": "true",
      "k8s-app": "etcd",
      "revision": "7"
    }
  },
  "timestamp": "2022-06-25T00:10:58Z",
  "window": "5m0s",
  "containers": [
    {
      "name": "etcd",
      "usage": {
        "cpu": "142m",
        "memory": "1197632Ki"
      }
    },
    {
      "name": "etcd-health-monitor",
      "usage": {
        "cpu": "42m",
        "memory": "34240Ki"
      }
    },
    {
      "name": "etcd-metrics",
      "usage": {
        "cpu": "28m",
        "memory": "17920Ki"
      }
    },
    {
      "name": "etcd-readyz",
      "usage": {
        "cpu": "8m",
        "memory": "31680Ki"
      }
    },
    {
      "name": "etcdctl",
      "usage": {
        "cpu": "0",
        "memory": "3200Ki"
      }
    },
    {
      "name": "POD",
      "usage": {
        "cpu": "0",
        "memory": "0"
      }
    }
  ]
}

List Pods on a Node

The following lists every Pod on a Node and outputs the namespace and pod name:

$ oc get pods -A -o json --field-selector spec.nodeName=worker-0.ocp-power.xyz | jq -r '.items[] | "\(.metadata.namespace) / \(.metadata.name)"'
openshift-cluster-node-tuning-operator / tuned-fdpl4
openshift-dns / dns-default-6vdcw

More posts