The Linux Pressure Stall Information, as part of the Control Group v2, provides an accurate accounting of a containers cpu, memory and io. The psi stats allow accurate and limited access to resources – no over-committing and no over-sizing.
However, it sometimes is difficult to see if the a container is being limited and could use more resources assigned.
This article is designed to help you diagnose and check your pods so you can get the best out of your workloads.
Check your workload
You can check the container in your Pod’s cpu.stat:
- Find the containerId
[root@cpi-c7b2-bastion-0 ~]# oc get pod -n test test-pod -oyaml | grep -i containerID
- containerID: cri-o://c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea
- Connect into the Pod.
[root@cpi-c7b2-bastion-0 ~]# oc rsh -n test test-pod
sh-4.4# find /sys -iname '*c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea*'
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4b90d9_20f9_427d_9414_9964f32379dc.slice/crio-c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea.scope
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4b90d9_20f9_427d_9414_9964f32379dc.slice/crio-conmon-c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea.scope
- Check the cpu.stat or io.stat or memory.stat.
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4b90d9_20f9_427d_9414_9964f32379dc.slice/crio-conmon-c050804396004e6b5d822541a58f299ea2b0e48936709175d6d57f3507cc6cea.scope/cpu.stat
usage_usec 11628232854
user_usec 8689145332
system_usec 2939087521
core_sched.force_idle_usec 0
nr_periods 340955
nr_throttled 8
throttled_usec 8012
nr_bursts 0
burst_usec 0
- We can see that the cpu is being throttled in
nr_throttled and throttled_usec. This is really a minor impact for a container.
nr_throttled 8
throttled_usec 8012
If the container had a higher number of throttled events, you want to check the number of cpus or memory that your container is limited to, such as:
nr_throttled 103
throttled_usec 22929315
- Check the container limits.
❯ NS=test
❯ POD=test-pod
❯ oc get -n ${NS} pod ${POD} -ojson | jq -r '.spec.containers[].resources.limits.cpu'
8
- Patch your Pod or update your application to increase the cpus.
Checking real-time stats
You can check the real-time stats top for your container pressure. Log on to your host.
find /sys/fs/cgroup/kubepods.slice/ -iname cpu.pressure | xargs -t -I {} cat {} | grep -v total=0
find /sys/fs/cgroup/kubepods.slice/ -iname memory.pressure | xargs -t -I {} cat {} | grep -v total=0
find /sys/fs/cgroup/kubepods.slice/ -iname io.pressure | xargs -t -I {} cat {} | grep -v total=0
This will show you all the pods that are under pressure.
for PRESSURE in $( find /sys/fs/cgroup/kubepods.slice/ -iname io.pressure)
do
if [ ! -z "$(cat ${PRESSURE} | grep -v total=0)" ]
then
if [ ! -z "$(cat ${PRESSURE} | grep -v "avg10=0.00 avg60=0.00 avg300=0.00")" ]
then
echo ${PRESSURE}
fi
fi
done
❯ cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde03ef16_000a_4198_9e04_ac96d0ea33c5.slice/crio-d200161683a680588c4de8346ff58d633201eae2ffd558c8d707c4836215645e.scope/io.pressure
some avg10=14.02 avg60=14.16 avg300=13.99 total=4121355556
full avg10=14.02 avg60=14.16 avg300=13.99 total=4121050788
In this case, I was able to go in and icnrease the total IO.
Tweak
You can tweak the cpu.pressure settings temporarily for a pod or system so the time used to evaluate is extended (this is the longest time possible).
The maximum window size is 10 seconds, and if you have kernel version less than 6.5 then the minimum window size is 500ms.
cat << EOF > /sys/fs/cgroup/cpu.pressure
some 10000000 10000000
full 10000000 10000000
EOF
Disabling psi in OpenShift
There are two methods to disable psi in OpenShift, the first is to set a kernel parameter, and the second is to switch from cgroupsv2 to cgroups.
Switch from cgroupsv2 to cgroups
You can switch from cgroupsv2 to cgroups – Configuring the Linux cgroup version on your nodes.
❯ oc patch nodes.config cluster --type merge -p '{"spec": {"cgroupMode": "v1"}}'
You’ll have to wait for each of the Nodes to restart.
Set the Kernel Parameter psi=0
In OpenShift, you can disable psi in using a MachineConfig.
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 99-worker-psi-disable
spec:
kernelArguments:
- psi=0
Check psi is enabled
You can check to see if it is enabled by checking one of the cpu.pressure, io.pressure or memory.pressure files. You’ll see “Operation not supported”.
sh-5.1# cat /sys/fs/cgroup/cpu.pressure
cat: /sys/fs/cgroup/cpu.pressure: Operation not supported
or
oc debug node/<node_name>
chroot /host
stat -c %T -f /sys/fs/cgroup
tmpfs
Summary
Linux PSI is pretty awesome. However, you should check your workload and verify it’s running correctly.
References
