I’ve had to run a number of queries for an OpenShift Cluster, Kubernetes and Linux recently, and here are my helpful queries:
Node Memory
If you want to check the Memory on each Node in your OpenShift Cluster, you can run the following oc command:
$ oc get nodes -o json | jq -r '.items[] | "\(.metadata.name) - \(.status.capacity.memory)"'
master-0.ocp-power.xyz - 16652928Ki
master-1.ocp-power.xyz - 16652928Ki
master-2.ocp-power.xyz - 16652928Ki
worker-0.ocp-power.xyz - 16652928Ki
worker-1.ocp-power.xyz - 16652928Ki
Node Memory Pressure
If you want to check the Memory usage on each Node in your OpenShift Cluster, you can run the following oc command:
Memory Pressure Per node:
$ oc adm top node --show-capacity=true
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master-0.ocp-power.xyz 1894m 15272Mi 93%
master-1.ocp-power.xyz 1037m 8926Mi 54%
master-2.ocp-power.xyz 1563m 10953Mi 67%
worker-0.ocp-power.xyz 1523m 6781Mi 41%
worker-1.ocp-power.xyz 933m 6746Mi 41%
Top Memory Usage per Pods
If you want to check the Top Memory Usage per Pod, you can run the following command:
$ oc adm top pod -A --sort-by='memory'
NAMESPACE NAME CPU(cores) MEMORY(bytes)
15% - openshift-kube-apiserver kube-apiserver-master-2.ocp-power.xyz 386m 2452Mi
11% - openshift-kube-apiserver kube-apiserver-master-0.ocp-power.xyz 225m 1924Mi
10% - openshift-kube-apiserver kube-apiserver-master-1.ocp-power.xyz 239m 1720Mi
List Container Memory Details per Pod
If you want to see the breakdown of Memory usage, you can use the following kubectl command:
$ kubectl top pod kube-apiserver-master-2.ocp-power.xyz -n openshift-kube-apiserver --containers
POD NAME CPU(cores) MEMORY(bytes)
kube-apiserver-master-2.ocp-power.xyz POD 0m 0Mi
kube-apiserver-master-2.ocp-power.xyz kube-apiserver514m 2232Mi
kube-apiserver-master-2.ocp-power.xyz kube-apiserver-cert-regeneration-controller 25m 51Mi
kube-apiserver-master-2.ocp-power.xyz kube-apiserver-cert-syncer0m 28Mi
kube-apiserver-master-2.ocp-power.xyz kube-apiserver-check-endpoints7m 41Mi
kube-apiserver-master-2.ocp-power.xyz kube-apiserver-insecure-readyz0m 16Mi
Checking the High and Low Memory Limits on a Linux Host
If you want to check the memory usage on a host in Gigabytes (including the max allocation), you can run the free command:
$ free -g -h -l
total used free shared buff/cache available
Mem: 15Gi 10Gi 348Mi 165Mi 5.2Gi 4.8Gi
Low: 15Gi 15Gi 348Mi
High: 0B 0B 0B
Swap: 0B 0B 0B
Use Observe > Metrics
If you have Metrics enabled, login to your OpenShift Dashboard, and click on Observe > Metrics and use one of the following
sum(node_memory_MemAvailable_bytes) by (instance) / 1024 / 1024 / 1024
:node_memory_MemAvailable_bytes:sum
Check Memory Usage on the CoreOS Nodes
If you want to check the memory details on each CoreOS node, you can use the following hack to SSH in and output the details.
$ for HN in $(oc get nodes -o json | jq -r '.items[].status.addresses[] | select(.type=="Hostname").address')
do
echo HOSTNAME: $HN
ssh core@$HN 'cat /proc/meminfo'
done
HOSTNAME: master-0.ocp-power-xyz
MemTotal: 16652928 kB
MemFree: 265472 kB
MemAvailable: 933248 kB
Buffers: 384 kB
Cached: 1387584 kB
SwapCached: 0 kB
Active: 688000 kB
Inactive: 7832192 kB
Active(anon): 120448 kB
Inactive(anon): 7307392 kB
Check Top in Batch on the CoreOS Nodes
If you want to check the Memory using Top (batch) on each CoreOS node, you can use the following hack to SSH in and output the details: (refer to link)
for HN in $(oc get nodes -o json | jq -r '.items[].status.addresses[] | select(.type=="Hostname").address')
do
echo
echo HOSTNAME: $HN
ssh core@$HN 'top -b -d 5 -n 1 -E g -o +%MEM'
sleep 10
done
HOSTNAME: master-0.ocp-power.xyz
top - 23:41:40 up 7 days, 11:58, 0 users, load average: 1.60, 2.24, 2.74
Tasks: 390 total, 1 running, 389 sleeping, 0 stopped, 0 zombie
%Cpu(s): 48.1 us, 10.6 sy, 0.0 ni, 39.4 id, 1.2 wa, 0.0 hi, 0.6 si, 0.0 st
GiB Mem : 15.9 total, 0.2 free, 14.3 used, 1.3 buff/cache
GiB Swap: 0.0 total, 0.0 free, 0.0 used. 0.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1018800 root 20 0 2661824 1.7g 21696 S 0.0 10.5 2896:40 kube-ap+
42247 root 20 0 11.3g 1.1g 14272 S 16.7 7.1 1483:55 etcd
1704 root 20 0 3984384 220480 31680 S 11.1 1.3 3289:24 kubelet
Pod Metrics (Thanks StackOverflow)
If you want to get raw cpu and memory metrics from OpenShift, you can run the following:
kubectl -n openshift-etcd get --raw /apis/metrics.k8s.io/v1beta1/namespaces/openshift-etcd/pods/etcd-master-0.ocp-power.xyz | jq
{
"kind": "PodMetrics",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"name": "etcd-master-0.ocp-power.xyz",
"namespace": "openshift-etcd",
"creationTimestamp": "2022-06-25T00:10:58Z",
"labels": {
"app": "etcd",
"etcd": "true",
"k8s-app": "etcd",
"revision": "7"
}
},
"timestamp": "2022-06-25T00:10:58Z",
"window": "5m0s",
"containers": [
{
"name": "etcd",
"usage": {
"cpu": "142m",
"memory": "1197632Ki"
}
},
{
"name": "etcd-health-monitor",
"usage": {
"cpu": "42m",
"memory": "34240Ki"
}
},
{
"name": "etcd-metrics",
"usage": {
"cpu": "28m",
"memory": "17920Ki"
}
},
{
"name": "etcd-readyz",
"usage": {
"cpu": "8m",
"memory": "31680Ki"
}
},
{
"name": "etcdctl",
"usage": {
"cpu": "0",
"memory": "3200Ki"
}
},
{
"name": "POD",
"usage": {
"cpu": "0",
"memory": "0"
}
}
]
}
List Pods on a Node
The following lists every Pod on a Node and outputs the namespace and pod name:
$ oc get pods -A -o json --field-selector spec.nodeName=worker-0.ocp-power.xyz | jq -r '.items[] | "\(.metadata.namespace) / \(.metadata.name)"'
openshift-cluster-node-tuning-operator / tuned-fdpl4
openshift-dns / dns-default-6vdcw
