July 2012

NUMA and Scale Up

Zhongke Chen

Confidential and Proprietary

SMP VS NUMA (NON-UNIFORM MEMORY ARCHITECTURE)

2Technology Education

SMPNUMA (!=MPP)

Confidential and Proprietary

HOW LINUX HANDLES NUMA

3

Default Policy: One process is fixed to one node.

Linux detects NUMA and deals it differently from SMP

Confidential and Proprietary

SCALE UP PROBLEM

What happens if one process uses large memory? (larger than the memory in one node)

4

Confidential and Proprietary

WHAT HAPPENS

5

Only memory on one node is used.

The system starts to swap heavily.

Confidential and Proprietary

WORKAROUNDS

numactl --interleave=all

(inefficient to access remote memory)

6

Confidential and Proprietary

TRUE SOLUTIONS

1.Multiple Smaller Processes

2.Control Policy via libnuma API

3.Another workaround: Virtualization

7

Confidential and Proprietary

REFERENCE

• http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access

• http://developer.amd.com/assets/LibNUMA-WP-fv1.pdf

• http://www.kernel.org/doc/Documentation/vm/numa_memory_policy.txt

• http://linux.die.net/man/8/numactl

• http://linux.die.net/man/3/numa

8