However, I had great difficulty starting the JVM with the (sparsely) documented settings. I could see that the OS allocated the expected amount of memory as huge-pages. However, starting the VM with the -XX:+UseLargePages option set always resulted in one of the following errors:
- Failed to reserve shared memory (errno = 28). // 'No space left on device'
When -Xmx is less than the huge page allocation:
- Failed to reserve shared memory (errno = 12). // 'Out of memory'
A helpful stackoverflow answer suggested that I apply another kernel parameter that is seemingly not documented in any of the standard -XX:+UseLargePages resources:
This parameter must be set to an equivalent size as /proc/sys/kernel/shmmax but expressed in 4KB pages units, so for a shmmax size of 25769803776 shmall should be 6291456. This solved some of my issues and I could test the correct instantiation of the JVM with:
- java -XX:+UseLargePages -Xmx20g -version
However, my application is executed by jsvc from Apache commons-daemon, and when starting it in this way the memory allocation problem remained. I hadn't expected this as the executing user had appropriate hugetlb group membership and adequate memlock limits in declared in /etc/security/limits.conf:
- appuser soft memlock 25165824
- appuser hard memlock 25165824
Now although jsvc is normally started by the root user, it switches to a configured exec user. Some experimentation showed that appuser's memlock limits where not applied when jsvc was switching from root to appuser and hence memory could not be allocated to the JVM. To get around this I also added root to the hugetlb group and provided some additional memlock limits for the root user:
- root soft memlock 25165824
- root hard memlock 25165824
Now everything works as expected.