Huge Page の解放について

ちょっと検証してみたメモ。

検証結果

  • vm.nr_hugepages を設定する
[root@yazekats-linux ~]# grep -i hugepage /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@yazekats-linux ~]# sysctl -w vm.nr_hugepages=200
vm.nr_hugepages = 200
[root@yazekats-linux ~]# grep -i hugepage /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:     200
HugePages_Free:      200
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
  • Huge Page を System V 共有メモリとして使うプログラムを実行する
[root@yazekats-linux hugepage_test]# gdb ./hugepage-shm
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/yazekats/Documents/github/yoheia/misc/hugepage_test/hugepage-shm...done.
(gdb) b 68
Breakpoint 1 at 0x40076c: file hugepage-shm.c, line 68.
(gdb) r
Starting program: /home/yazekats/Documents/github/yoheia/misc/hugepage_test/hugepage-shm 
shmid: 0x25002c

Breakpoint 1, main () at hugepage-shm.c:68
68		if (shmaddr == (char *)-1) {
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.x86_64
(gdb) 
  • Huge Page が割り当てられている
[yazekats@yazekats-linux ~]$ grep -i hugepage /proc/meminfo 
AnonHugePages:         0 kB
HugePages_Total:     200
HugePages_Free:      200
HugePages_Rsvd:      128 ★
HugePages_Surp:        0
Hugepagesize:       2048 kB
HugePages_Total is the size of the pool of huge pages.
HugePages_Free  is the number of huge pages in the pool that are not yet
                allocated.
HugePages_Rsvd★  is short for "reserved," and is the number of huge pages for
                which a commitment to allocate from the pool has been made,
                but no allocation has yet been made.  Reserved huge pages
                guarantee that an application will be able to allocate a
                huge page from the pool of huge pages at fault time.
HugePages_Surp  is short for "surplus," and is the number of huge pages in
                the pool above the value in /proc/sys/vm/nr_hugepages. The
                maximum number of surplus huge pages is controlled by
                /proc/sys/vm/nr_overcommit_hugepages.
  • 実行したプログラムのPIDを調べる
[yazekats@yazekats-linux ~]$ ps auxw|grep [h]ugepage-shm
root      2509  0.0  0.0 187884 12928 pts/0    S+   12:09   0:00 gdb ./hugepage-shm
root      2524★  0.0  0.0   3972   284 pts/0    t    12:10   0:00 /home/yazekats/Documents/github/yoheia/misc/hugepage_test/hugepage-shm
  • PID:2524 が作成した共有メモリセグメントの shmid を調べる。
[root@yazekats-linux ~]# ipcs -p|grep 2524
2424876★    root       2524       2524      
  • shmid:2424876 の共有メモリセグメントを確認する。
[root@yazekats-linux ~]# ipcs -m |grep 2424876
0x00000002 2424876    root       600        2684354561 

268435456 bytes = 256MB使っている

  • プログラムを終了する
(gdb) quit
A debugging session is active.

	Inferior 1 [process 2524] will be killed.

Quit anyway? (y or n) y
  • プロセスがないことを確認する。
[yazekats@yazekats-linux ~]$ ps auxw|grep [h]ugepage-shm
[yazekats@yazekats-linux ~]$
  • Huge Page は解放されていない。
[yazekats@yazekats-linux ~]$ grep -i hugepage /proc/meminfo 
AnonHugePages:         0 kB
HugePages_Total:     200
HugePages_Free:      200
HugePages_Rsvd:      128
HugePages_Surp:        0
Hugepagesize:       2048 kB
  • 共有メモリも解放されない。
[root@yazekats-linux ~]# ipcs -p|grep 2524
2424876    root       2524       2524      
[root@yazekats-linux ~]# ipcs -m |grep 2424876
0x00000002 2424876    root       600        268435456  0     
  • ipcrm コマンドで共有メモリを解放する
[root@yazekats-linux ~]# ipcrm -m 2424876
  • Huge Page も共有メモリも解放されている
[root@yazekats-linux ~]# grep -i hugepage /proc/meminfo 
AnonHugePages:         0 kB
HugePages_Total:     200
HugePages_Free:      200
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@yazekats-linux ~]# ipcs -p|grep 2524
[root@yazekats-linux ~]# ipcs -m |grep 2424876

参考

hugepage-shm.c のソースコード
/*
 * hugepage-shm:
 *
 * Example of using huge page memory in a user application using Sys V shared
 * memory system calls.  In this example the app is requesting 256MB of
 * memory that is backed by huge pages.  The application uses the flag
 * SHM_HUGETLB in the shmget system call to inform the kernel that it is
 * requesting huge pages.
 *
 * For the ia64 architecture, the Linux kernel reserves Region number 4 for
 * huge pages.  That means that if one requires a fixed address, a huge page
 * aligned address starting with 0x800000... will be required.  If a fixed
 * address is not required, the kernel will select an address in the proper
 * range.
 * Other architectures, such as ppc64, i386 or x86_64 are not so constrained.
 *
 * Note: The default shared memory limit is quite low on many kernels,
 * you may need to increase it via:
 *
 * echo 268435456 > /proc/sys/kernel/shmmax
 *
 * This will increase the maximum size per shared memory segment to 256MB.
 * The other limit that you will hit eventually is shmall which is the
 * total amount of shared memory in pages. To set it to 16GB on a system
 * with a 4kB pagesize do:
 *
 * echo 4194304 > /proc/sys/kernel/shmall
 */

#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/mman.h>

#ifndef SHM_HUGETLB
#define SHM_HUGETLB 04000
#endif

#define LENGTH (256UL*1024*1024)

#define dprintf(x)  printf(x)

/* Only ia64 requires this */
#ifdef __ia64__
#define ADDR (void *)(0x8000000000000000UL)
#define SHMAT_FLAGS (SHM_RND)
#else
#define ADDR (void *)(0x0UL)
#define SHMAT_FLAGS (0)
#endif

int main(void)
{
	int shmid;
	unsigned long i;
	char *shmaddr;

	shmid = shmget(2, LENGTH, SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W);
	if (shmid < 0) {
		perror("shmget");
		exit(1);
	}
	printf("shmid: 0x%x\n", shmid);

	shmaddr = shmat(shmid, ADDR, SHMAT_FLAGS);
	if (shmaddr == (char *)-1) {
		perror("Shared memory attach failure");
		shmctl(shmid, IPC_RMID, NULL);
		exit(2);
	}
	printf("shmaddr: %p\n", shmaddr);

	dprintf("Starting the writes:\n");
	for (i = 0; i < LENGTH; i++) {
		shmaddr[i] = (char)(i);
		if (!(i % (1024 * 1024)))
			dprintf(".");
	}
	dprintf("\n");

	dprintf("Starting the Check...");
	for (i = 0; i < LENGTH; i++)
		if (shmaddr[i] != (char)i) {
			printf("\nIndex %lu mismatched\n", i);
			exit(3);
		}
	dprintf("Done.\n");

	if (shmdt((const void *)shmaddr) != 0) {
		perror("Detach failure");
		shmctl(shmid, IPC_RMID, NULL);
		exit(4);
	}

	shmctl(shmid, IPC_RMID, NULL);

	return 0;
}
hugepage-shm.c のコンパイル手順
[yazekats@yazekats-linux hugepage_test]$  gcc -g -o hugepage-shm hugepage-shm.c
[yazekats@yazekats-linux hugepage_test]$  ls -1
hugepage-shm ★←実行可能ファイルができる
hugepage-shm.c
hugepage-shm.c の実行手順
[root@yazekats-linux hugepage_test]# ./hugepage-shm 
shmid: 0x22802c
shmaddr: 0x7f8aa5600000
Starting the writes:
................................................................................................................................................................................................................................................................
Starting the Check...Done.

環境

[root@yazekats-linux ~]# cat /etc/issue
Oracle Linux Server release 6.4
Kernel \r on an \m

[root@yazekats-linux ~]# uname -r
2.6.39-400.17.1.el6uek.x86_64


補足(2014/12/14):
vm.nr_hugepages を 0 にしても解放されません。

[root@yazekats-linux ~]# sysctl -w vm.nr_hugepages=0
vm.nr_hugepages = 0
[root@yazekats-linux ~]# grep -i hugepage /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:     128
HugePages_Free:      128
HugePages_Rsvd:      128
HugePages_Surp:      128
Hugepagesize:       2048 kB
[root@yazekats-linux ~]# tail /etc/sysctl.conf
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

vm.nr_hugepages=0

[root@yazekats-linux ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
vm.nr_hugepages = 0
[root@yazekats-linux ~]# grep -i hugepage /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:     128
HugePages_Free:      128
HugePages_Rsvd:      128
HugePages_Surp:      128
Hugepagesize:       2048 kB