Paul Bastide

Help… My SystemMemoryExceedsReservation

Written by

Paul Bastide

in

,

Red Hat explains the alert in SystemMemoryExceedsReservation alert received in OCP 4. There is also some detail in alerts/machine-config-operator/SystemMemoryExceedsReservation.md.

a warning triggered when the *memory usage* of the *system processes* exceeds the 95% of the reservation, not the total memory in the node.

You can check your configuration by ssh’ing to one of the workers, and sudo ps -ef | grep /usr/bin/kubelet | grep system-reserved

[root@worker-0 core]# sudo ps -ef | grep  /usr/bin/kubelet | grep system-reserved
root        2733       1 15 Nov04 ?        12:21:38 /usr/bin/kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-cgroups=/system.slice/crio.service --node-labels=node-role.kubernetes.io/worker,node.openshift.io/os_id=rhel, --node-ip=10.20.29.240 --minimum-container-ttl-duration=6m0s --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --cloud-provider= --hostname-override= --provider-id= --pod-infra-container-image=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d2f23cbaebe30a59f7af3b5a9e9cf6157f8ed143af494594e1c9dcf924ce0ec --system-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi --v=2

You’ll notice the default is a half core and 1G memory cpu=500m,memory=1Gi.

You can tweak the configuration using:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: set-allocatable
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    systemReserved:
      cpu: 1000m
      memory: 3Gi

Wait until the restart 99% sure that it just restarts kubelet in 4.19, without a reboot.

More posts