con1855_chambille-operating a 16tb jvm

8/17/2019 Con1855_chambille-Operating a 16tb Jvm

1/27

Antoine ChambilleHead of Research & Development, Quartet FS

@activepivot

Amir JavanshirPrincipal Software Engineer, racle

@!mir"#avanshir


2/27

w w w . q u a r t e t f s . c o m$

%redit Ris 'se %ase

R

( S

)

E

*

+

( *

E

S

!

%

,

( -

E P

( -

,

-aluation of .nancialinstruments for eachsimulation

/ 01trades

/ $22time points

1onte3%arlo simulationof future maret data

/ 4222

5 6 7 of raw data

Performcalculation o

!ggregate tdistri8


3/27

w w w . q u a r t e t f s . c o m9

In-Memory Database…

%olumn Store

High %ardinalit: 7itmap(nde/

1ultidimensional!ggregation

Designed for 1ulticore

!ctivePivot (n31emor: !nal:tics

… in Pure Java

Strong customi;ationcapa8ilities

Fast Safe !ccessi8le to man:


4/27

%op:right < $20= racle and

Oracle platform for in-memory comp1B 7ig 1emor: 1achineC era8:te Scale %omputing levStandards

HE platform for in3me

computing3 !"#A$%!" SSE1 1E1R

3!"#A$%!"&P"# &"CO'D

&%&!"M$A'D(ID!)

*+,!"#A$%!"&P"# &"CO'D

M"MO#% $A'D(ID!)

P

$

3.,CO#"& 3/0COMP1!"!)#"AD&

) i g h A v a i l a b i l i t y

$ u

i l 2

i n

O( atency 4 *0/ns


5/27 w w w . q u a r t e t f s . c o m4

3Heap 1emor: java.nio.Bufer G

// Allocate an array of doubles (in the heap)

double[] array = new double[1024*1024]

// !!!

double "alue#4 = array[#4]

array[#00] = array[100] $ array[200]

// Allocate an off%heap buffer of doubles

&ouble'uffer buf = 'yte'uffer!allocateDirect(*1024*1024)!as&ouble'uffer()

// !!!

double "alue#4 = buf!et(#4)

buf!put(#00 buf!et(100) $ buf!et(200))


6/27 w w w . q u a r t e t f s . c o mB

3Heap 1emor: sun.misc.UnsaeG

// +ou ,now what this is

sun!-isc!.nsafe .A = getUnsafe()

// Allocate 3' of off%heap -e-ory

lon pointer = .A!allocateMemory(*1024*1024)

.A!setMemory(pointer *1024*1024 (byte) 0)

// !!!

double "alue#4 = .A!getDouble(pointer $ *#4)

.A! putDouble( pointer $ *#00

.A!getDouble(pointer $ *100) $ .A!getDouble(pointer $ *200)

)


7/27


Performance5

3Heap 1emor:in the core data structures

%olumns of num8ers

-ectors of simulations

Hash ta8les

(nde/es Seriali;ed o8ects

3Heap 1emor:

1sability5

Standard #ava in the Iuengine

* to , ratio

9 7 Heap for Iueriescalculations

0$ 7 dataset 3HeaJ ld L generation

J oung L generation


8/27


6or7Join Pool 8


9/27


'nleashing Many3%ores Parallelism

7efore !fter

%P'

D!!

D


10/27



* threa2

< 3/0


11/27



Quer: ime secondsG

02

$2

92

2

*3 s

*0 s

, s

& 3<

* !$

& *=<

= !$

& .<

3 !$


12/27


*'1! !rchitecture


13/27


*'1! !rchitecture

9$ *'1!nodes


14/27


*'1! !rchitecture

!llocated memor: !

Process !

Process 7

!llocated memor: 7


15/27


*'1! S:stem Ni8rar:

protected interface u-aibrary e5tends co-!sun!6na!ibrary 7

/** 8return %1 if .3A is not a"ailable 9 */

int nu-a:a"ailable()

/** 8return the nu-ber of -e-ory nodes in the syste-! */

int nu-a:nu-:confiured:nodes()

/** 8return the .3A node the cpu currently belons to! */

int nu-a:node:of:cpu(int cpu)

int nu-a:run:on:node(int node)

int nu-a:distance(int node1 int node2)

//!!!

;

Linux


16/27

w w w . q u a r t e t f s . c o m0B

*'1! S:stem Ni8rar:

u-aibrary nu-aib = (u-aibrary) ati"e!loadLibrary(


17/27

w w w . q u a r t e t f s . c o m0O

*'1! S:stem Ni8rar:

u-aibrary nu-aib = (u-aibrary) ati"e!loadLibrary(


18/27


'nleashing *'1! Parallelism

D!! P!R((*S


19/27


'nleashing *'1! Parallelism

Quer: hroughput

$22

=22

B22

2

& 3<

* !$

& *=<

= !$

& .<

3 !$

=>?$s

3>>?$s

@,>?$s


20/27

w w w . q u a r t e t f s . c o m$2

+ar8age %ollection (G1 and the 3TB Hea!

%CCD$.se1?

%CCD>nitialEeapiFe=G2H#G4GG2 %CCD3a5EeapiFe=G2H#G4GG2

%CCD3a5&irect3e-oryiFe=1G1H41GH#GGG12%CCD%.seAdapti"eiFeolicy %CCD$.nloc,5peri-entalI3@ptions

%CCDur"i"orJatio=1

%CCDarallel?Khreads=HL0 %CCD?onc?Khreads=240 %CCD1EeapMasteercent=#

%CCD13a5ewiFeercent=# %CCD13i5ed?i"eKhresholdercent=0

%CCD>nitiatinEeap@ccupancyercent=10 %CCD1ewiFeercent=N0

%CCD1Jeser"eercent=10 %CCD1EeapJeioniFe=GG##44G2

%CCDareaeiFe>n'ytes=2L4G#4#L %CCD3a5?ause3illis=1000

%CCDJeser"ed?ode?acheiFe=2#1L#240

%CCD$rintAdapti"eiFeolicy %CCD$rint? %CCD

$rint?Application?oncurrentKi-e

%CCD$rint?ApplicationtoppedKi-e %CCD$rint?&etails %CCD

$rint?Ki-eta-ps

%CCD$rintKenurin&istribution %CCD$e-ented?ode?ache

%CCD1oe"el=finest


21/27



%CCD$.se1?



%CCDur"i"orJatio=1










$rint?Ki-eta-ps


%CCD1oe"el=finest


22/27

w w w . q u a r t e t f s . c o m$$


%CCD$.se1?



%CCDur"i"orJatio=1










$rint?Ki-eta-ps


%CCD1oe"el=finest

8 ll i


23/27



%CCD$.se1?



%CCDur"i"orJatio=1










$rint?Ki-eta-ps


%CCD1oe"el=finest

+ 8 % ll i


24/27

w w w . q u a r t e t f s . c o m$=


%CCD$.se1?



%CCDur"i"orJatio=1










$rint?Ki-eta-ps


%CCD1oe"el=finest

+ 8 % ll ti St t


25/27


+ar8age %ollection Stress est

+ar8age %ollection ime Distri8ution

Mh t * t


26/27

w w w . q u a r t e t f s . c o m$B

Alrea2y:or7ing HugeG read3onl:

Iuerieson a lot ofG static data

Mhats *e/t

!he roa2 toOperational Analytics Narge data updates

Real3time data updates

%ontinuous Iueries on live data

J 1i/ed Morload L operations

Oracle an2 uarpartnership racle Engineering will u

!ctivePivot as the referefor ver: large heap supp

Quartet FS will implemerecommendations 8: rEngineering

ObBectives Scale +% on man: cores

*'1!

Reduce +% pause times

!ccelerate Full +%

Support 1i/ed3Morload


27/27

Do :ou want to 8e paPlease mae the c

w w w I u a r t e t f s c o m

Antoine Chambilleantoinecham8ille@Iuartetfscom

Amir Javanshiramiravanshir@oraclecom

con1855_chambille-operating a 16tb jvm

Documents