dr. low latency or: how i learned to stop worrying about pauses and love the memory

© C2B2 Consulting Limited 2013

All Rights Reserved

@author Jaromir Hamala, [email protected] @author(type=Twitter) @jerrinot

Dr. Low Latency or: How I Learned to Stop Worrying about Pauses and Love the

Memory Heap

mailto:[email protected]


All Rights Reserved

Who Am I?

■ Jaromir Hamala

■ A curious developer

■ Who likes to break play with stuff


All Rights Reserved

Safe Harbor

This talk may contain non-sense!


All Rights Reserved

Facts

• Memory is Cheap

• We have a lot of data

• Data in Memory == Fastest Possible Data (apart from CPU

caches)

http://www.npr.org/blogs/thetwo-way/2012/06/06/154416480/ho-hum-dull-and-boring-are-now-a-pair


All Rights Reserved

On-Heap Allocation

• CoolObject cool = new CoolObject()

• Great:

– It’s easy and usually *very* fast! • Allocation can be actually faster than in C

– GC goodies!

• Not so great:

– GC leads to Stop-the-World Pauses

– Bigger Heap -> Longer Pauses


All Rights Reserved

Why to go Off-Heap?

• Latency caused by GC

• Locality

• Sharing with non-Java

http://digboston.com/boston-lulz/2013/08/trolley-trollop-first-comes-love/attachment/why-god-why-kitten1/


All Rights Reserved

GC Implementations

• (Old Gen) GC in HotSpot (Oracle/Sun JDK, OpenJDK)

– ParOld – STW pauses by design!

– CMS – fragmentation -> STW pause

– G1 – Immature (yet?), not fulfilling the expectations -> occasional STW pauses


All Rights Reserved

Latency can kill you!


All Rights Reserved

What is Off-Heap?

• Area of memory not maintained by GC

– Manual memory control

• No Notion of Java object

– serialization

http://nickshell1983.wordpress.com/2010/06/29/taking-a-god-nudged-leap-of-faith/


All Rights Reserved

Off-Heap Options?

• Direct Memory Buffer

• sun.misc.Unsafe

• JNI

• Memory Mapped File

• ….

http://en.wikipedia.org/wiki/File:Road_to_Hell_One_Sheet.jpg


All Rights Reserved

Direct Memory Buffer

• ByteBuffer directBuf = ByteBuffer.allocateDirect(int noBytes);

• Part of Java NIO

• Max. 2GB / buffer

• Pure Java = Portable

• getX()/putX() methods


All Rights Reserved

sun.misc.Unsafe

– long addr = unsafe.allocateMemory(noBytes)

– Not a standardized part of JDK!

– Implemented by most JVM vendors

• Sun / Oracle / IBM

– It can disappear from JDK without warning!


All Rights Reserved

Cassandra I.

public class NativeAllocator implements IAllocator

{

static final Unsafe unsafe;

static {

try {

Field field = sun.misc.Unsafe.class.getDeclaredField("theUnsafe");

field.setAccessible(true);

unsafe = (sun.misc.Unsafe) field.get(null);

}

catch (Exception e) {

throw new AssertionError(e);

}

}

public long allocate(long size) {

return unsafe.allocateMemory(size);

}

public void free(long peer) {

unsafe.freeMemory(peer);

}

}


All Rights Reserved

JNI

• Java Native Interface

• Native library means worst portability between platforms

• JNI Overhead

• Can re-use already existing (native) allocator


All Rights Reserved

Cassandra II.

• Java Native Interface via JNA public class JEMallocAllocator implements IAllocator {

public interface JEMLibrary extends Library {

long malloc(long size);

void free(long pointer);

}

private final JEMLibrary library;

public JEMallocAllocator() {

library = (JEMLibrary) Native.loadLibrary("jemalloc", JEMLibrary.class);

}

public long allocate(long size) {

return library.malloc(size);

}

public void free(long peer) {

library.free(peer);

}

}


All Rights Reserved

Memory Mapped File

• Part of NIO

• Pure Java -> Portable

• Allows sharing data between processes

• Persisted by OS


All Rights Reserved

private static int count = 1010241024; //10 MB

public static void main(String[] args) throws Exception {

RandomAccessFile memoryMappedFile = new RandomAccessFile("largeFile.txt", "rw");

//Mapping a file into memory

MappedByteBuffer out = memoryMappedFile.getChannel().map(FileChannel.MapMode.READ_WRITE, 0, count);

//Writing into Memory Mapped File

for (int i = 0; i < count; i++) {

out.put((byte) 'A');

}

System.out.println("Writing to Memory Mapped File is completed");

//reading from memory file in Java

for (int i = 0; i < 10 ; i++) {

System.out.print((char) out.get(i));

}

System.out.println("Reading from Memory Mapped File is completed");

}

Javin Paul,

http://javarevisited.blogspot.com/2012/01/memorymapped-file-and-io-in-java.html


All Rights Reserved

Use cases for Off-Heap?

• Caches / Data Grids

– A lot of data

– Objects with well defined lifecycle

• (Extremely) Latency-Sensitive tasks

– High-Frequency trading…

• Sharing Data with non-Java


All Rights Reserved

Apache DirectMemory

• “…is a off-heap cache for the Java Virtual Machine”

• Easy to integrate

• Friendly License

• Experimental

• http://directmemory.apache.org/

http://directmemory.apache.org/

http://directmemory.apache.org/


All Rights Reserved

DirectMemory Usage

CacheService<Object, Object> cacheService =

new DirectMemory<Object, Object>()

.setNumberOfBuffers( 10 )

.setSize( 1000 )

.setInitialCapacity( 100000 )

.setConcurrencyLevel( 4 )

.newCacheService();

put( K key, V value )

put( K key, V value, int expiresIn )


All Rights Reserved

Is Off-Heap The Future of Java?

• Oh God! Please no!

• R&D should be aimed to more effective GC

– Zing JVM (based on HotSpot, C4 - the fully concurrent GC)

– RedHat Shenandoah?


All Rights Reserved

Final Warning(s)

• Stay On-Heap unless you have no other choice!

• If you think you have no other choice, think twice

• Do NOT re-invent the wheel! (Memory Management)

• Measure, measure, measure!


All Rights Reserved

Further Sources

• http://mechanical-sympathy.blogspot.com/ – Martin Thompson. There is a brilliant mailing list as well!

• http://psy-lob-saw.blogspot.com – Nitsan Wakart

• http://vanillajava.blogspot.com – Peter Lawrey

• http://insightfullogic.com/blog/ – Richard Warburton

• http://ashkrit.blogspot.com – Ashkrit

http://mechanical-sympathy.blogspot.com/




http://psy-lob-saw.blogspot.com/






http://vanillajava.blogspot.com/



http://insightfullogic.com/blog/

http://insightfullogic.com/blog/

http://ashkrit.blogspot.com/



dr. low latency or: how i learned to stop worrying about pauses and love the memory

Technology

os c2b2 consulting limited

stuff c2b2 consulting

memory file

memory heap

unsafe jni memory mapped

memory mappedbytebuffer

facts memory

area of memory