This demonstration shows the Hugepages allocation in OpenShift and some of the finer debug points. You’ll find the example data at link
Note: The allocated Hugepages memory is not deallocated.
- Login to OpenShift
$ oc login
- List the Nodes
$ oc get nodes -l node-role.kubernetes.io/worker
NAME STATUS ROLES AGE VERSION
lon06-worker-0.xip.io Ready worker 9d v1.23.5+3afdacb
lon06-worker-1.xip.io Ready worker 9d v1.23.5+3afdacb
- Check one of your worker nodes by starting a terminal session
$ ssh core@lon06-worker-0.xip.io
- Check the Hugepagesize to verify it exists
$ grep Hugepagesize /proc/meminfo
Hugepagesize: 16384 kB
- Check the
vm.nr_hugepages
, if it’s zero we’ll need to set it up.
$ sysctl vm.nr_hugepages
vm.nr_hugepages = 0
- You can manually set until reboot (you’ll need to do this one each worker) or use a MachineConfig
$ sudo sysctl -w vm.nr_hugepages=256
vm.nr_hugepages = 256
- Use a MachineConfig to set
vm.nr_hugepages
$ oc apply -f machineconfig.yaml
machineconfig.machineconfiguration.openshift.io/99-sysctl-nr-hugepages created
- Wait for an update to the MachineConfig
$ oc wait mcp/worker --for condition=updated --timeout=25m
- Create the Hugepages demonstration pod
$ oc apply -f pod.yaml
pod/hugepages-demo created
- Check the hugepages output is correct. You are looking for Page Size 16M and total size.
$ oc exec -it hugepages-demo -- grep hugepages/demo /proc/1/smaps -A3
7efff8000000-7f0000000000 rw-s 00000000 00:1de 197766 /dev/hugepages/demo
Size: 131072 kB
KernelPageSize: 16384 kB
MMUPageSize: 16384 kB
- Find the node the pod is running on:
$ oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hugepages-demo 1/1 Running 0 5m30s 10.128.2.12 lon06-worker-1.xip.io <none> <none>
- Check the HugePages value.
$ grep HugePages_ /proc/meminfo
HugePages_Total: 20
HugePages_Free: 17
HugePages_Rsvd: 5
HugePages_Surp: 0
- Check the Allocatable HugePages allocation is non-zero
$ oc get node lon06-worker-1.xip.io -o jsonpath="{.status.allocatable}" | jq -r .
{
"cpu": "7500m",
"ephemeral-storage": "115586611009",
"hugepages-16Gi": "0",
"hugepages-16Mi": "4Gi",
"memory": "28069952Ki",
"pods": "250"
}
-
Check that there are allocated Hugepages (on one of the Worker nodes)
-
Switch to the root user
$ sudo -s
- Find the hugepaged process
ps -ef | grep hugepaged
- Verify the allocated data
$ PROC=27693
$ grep -A3 'demo' /proc/${PROC}/smaps
7efff8000000-7f0000000000 rw-s 00000000 00:1de 197766 /dev/hugepages/demo
Size: 131072 kB
KernelPageSize: 16384 kB
MMUPageSize: 16384 kB
Summary
You have seen how to configure a node to support hugepages and deploy a Pod with Hugepages support, and confirm Hugepages are used.
<hr>
References
- Red Hat OpenShift 4.10: How huge pages are consumed by apps
- Red Hat Customer Portal: [RHEL] How do I check for hugepages usage and what is using it?
- Red Hat Customer Portal: How to use, monitor, and disable transparent hugepages in Red Hat Enterprise Linux 6 and 7?
- Povilas: Go Memory Management
- Red Hat Universal Base Image 8
- Linux: How to force any application to use Hugepages without modifying the source code
- Kernel: HugeTLB Pages
- Linux Kernel: SelfTests Hugepage example using mmap
- StackOverflow: Why mmap cannot allocate memory?
- Erik Rigtorp: Using huge pages on Linux
Appendix: Install Hugepages Tools
- Check that hugectl is provided by your package manager
yum whatprovides hugectl
- Install the Hugectl tools
yum -y install libhugetlbfs-utils libhugetlbfs
Appendix: Install Build Tools
The minimum build tools required to build this sample project are make
and golang
.
yum install -y golang make
Appendix: mmap allocation issues
If you see mmap: Cannot allocate memory
, then you should run echo 20 > /proc/sys/vm/nr_hugepages
which switches the hugepages to non-zero.
Appendix: Using Tuned to Configure the Node with Alternative Page Sizes
Create the Tuned configuration and wait for a node restart.
- Create a tuned.yaml file with the additional page size and the number of pages per the manifests/tuned.yaml
cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=50
- Apply the configuration
$ oc apply -f manifests/tuned.yaml
tuned/hugepages created
- Wait while the node is restarted and have fun.
Appendix: huge tools
-
Install
yum install libhugetlbfs-utils -y
-
Check the mounts
$ hugeadm --list-all-mounts
Mount Point Options
/dev/hugepages rw,seclabel,relatime,pagesize=16M
/var/lib/hugetlbfs/pagesize-16MB rw,seclabel,relatime,pagesize=16M
/var/lib/hugetlbfs/pagesize-16GB rw,seclabel,relatime,pagesize=16384M
- Check the Pools and what setting they have
$ hugeadm --pool-list
Size Minimum Current Maximum Default
16777216 20 20 20 *
17179869184 0 0 0
- Start an application written with glibc to transparently use hugepages.
$ hugectl myapp
Note, it does not work with non-glibc apps (e.g. golang)
- Create the mounts automatically:
$ hugeadm --create-mounts
And then you can check the mounts
$ mount | grep pagesize
none on /var/lib/hugetlbfs/pagesize-16MB type hugetlbfs (rw,relatime,seclabel,pagesize=16M)
none on /var/lib/hugetlbfs/pagesize-16GB type hugetlbfs (rw,relatime,seclabel,pagesize=16384M)
Appendix: Manual Creation of the Hugepages FS
To mount 64KB pages (if the system hardware supports it):
mkdir -p /mnt/hugetlbfs-16K
mount -t hugetlbfs none -opagesize=16777216 /mnt/hugetlbfs-16K
mkdir /dev/hugepages16G
mount -t hugetlbfs -o pagesize=17179869184 none /dev/hugepages16G
Appendix: Check Hugepage Sizes
Check Hugepage sizes
$ ls /sys/kernel/mm/hugepages
hugepages-16384kB hugepages-16777216kB
Appendix: List sysctl settings
$ sysctl -a | grep hugepages
vm.nr_hugepages = 642
vm.nr_hugepages_mempolicy = 642
vm.nr_overcommit_hugepages = 676
To allocate our 2048 Huge Pages we can use:
$ echo 2048 > /proc/sys/vm/nr_hugepages
To disable transparent huge pages
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Appendix: Check Hugepage Mount usage for a process
- Check Hugepage Mount usage for a process (you need to know the process id and the filesystem name)
$ PROC=4991
$ grep -A3 '/var/lib/hugetlbfs/pagesize-16MB/demo' /proc/${PROC}/smaps
7efff8000000-7f0000000000 rw-s 00000000 00:2f 63535 /var/lib/hugetlbfs/pagesize-16MB/demo
Size: 131072 kB
KernelPageSize: 16384 kB
MMUPageSize: 16384 kB
Is this a Red Hat or IBM supported solution?
No. This is only a proof of concept that serves as a good starting point to understand how HugePages works in OpenShift.