�

Lijie  Xu1,2,  Jie  Liu1,  and  Jun  Wei1 1Institute  of  Software,  Chinese  Academy  of  Sciences

2University  of  Chinese  Academy  of  Sciences

FMEM:  A  Fine-­‐grained  Memory  Estimator  for  MapReduce  Jobs �

6/26/2013

       Outline �

n  What  is  FMEM?  n  Why  do  we  develop  it?  n  Is  it  a  hard  problem?  n  How  to  develop  it?  n  How  to  estimate  the  fine-­‐grained  memory  usage?  n  Evaluation  n  Related  work  n  Conclusion

       What  is  FMEM? �

n  It  is  designed  to  analyze,  predict  and  optimize  the  fine-­‐grained  memory  usage  of  map  and  reduce  tasks.  

   

       Why  do  we  want  to  develop  FMEM? �

n  For  users    •  feel  hard  to  specify  the  memory-­‐related  

configurations.  •  e.g.,  buffer  size,  reducer  number,  Xmx,  Xms,  and  etc.  

•  OutOfMemory  error  or  performance  degradation  

n  For  task  schedulers    •  e.g.,  YARN  and  Mesos  •  they  allocate  resource  and  schedule  tasks  according  to  

the  CPU  and  memory  requirement  of  mappers/reducers.  

n  For  Hadoop  committers    •  design  a  more  memory-­‐efficient  framework.  

       So,  what  is  the  fundamental  problem? �

n  Tell  job’s  memory  usage  before  running  it  on  big  dataset.  n  f  (Data  d,  Conf  f,  Program  p,  Resource  r)              à  memory  usage  mu  of  mappers  and  reducers  n  Add  <SampData  sd,  SampConf  sf,  SampProfiles  sp>            à  memory  usage  mu  of  real  mappers  and  reducers  

FMEM  

Big  data  

sample   Sample  job  

profiles  

big  job  profiles  

sampConf  

Conf  

sampling  Dataflow&  memory  usage  

   Is  it  a  hard  problem? �map()          {      }  reduce()  {      }  Pig  /HiveQL  

Data  Configura<on  

User  layer

Framework    layer

Execution  layer

Memory

JVM

disk

   How  to  do  it?  dataflowàobjectsàusage   �map()          {      }  reduce()  {      }  Pig  /HiveQL  

Data  Configura<on  

User  layer

Framework    layer

Execution  layer

Memory

JVM

disk Dataflow  volume  

In-­‐memory  objects’  size  

rules  of  memory  usage  

memory  usage  

 How  to  do  it?  dataflowàobjectsàusage �

n  Built-­‐in  Monitor  •  get  Records/Bytes  Statistics  &  real-­‐time  memory  usage  

n  Profiler  •  count  intermediate  data  and  TempObjs  in  each  phase  

Dataflow  volume  

Instrument  get  sta<s<cs  

   How  to  do  it?  dataflowàobjectsàusage   �

n  Estimate  the  dataflow  •  We  actually  build  a  MapReduce  simulator  to  estimate  

the  intermediate  data  under  specific  <Data  d,  Conf  f,  sample  profiles  sp>.

Dataflow  volume  

     Which  are  memory-­‐consuming  objects? �

n  Memory  buffer  (configuration)  •  e.g.,  spill  buffer  (io.sort.mb)  and  user-­‐defined  arrays  

n  <K,  V>  records  (dataflow)  •  map()  outputs  records  into  spill  buffer,  in-­‐memory  segments  in  shuffle  buffer  

n  Temporary  objects  (dataflow  and  programs)  •  byproducts  of  records  such  as  char[],  byte[],  String  and  ArrayList  •  A  WordCount  mapper  produces  480MB  java.nio.HeapCharBuffer  objects  with  64MB  

input  split.  

n  Others  •  code  segment,  native  libraries,  call  stack  and  etc.

In-­‐memory  objects’  size  

Input  Split

map()

kvbuffer

OnDiskSeg

OnDiskSeg

OnDiskSeg

merge  &

 sort

reduce()

Output

HTTPfetch

kvoffsetskvindices

partition

Segment

InMemSeg The  SameMemory  Buffer

ShufBuf

SpillBuffer

partition,sort, combine,

spill to disk

mergeon disk

Other  Mappers Other  Reducers

shuffle, combine, in-memory & on-disk merge

Map  &  Spill  Phase Merge  Phase Shuffle  Phase Sort  Phase Reduce  Phase

Spill  piece

   How  to  do  it?  dataflowàobjectsàusage   �

memory  usage

in-­‐memory  objects

dataflow  volume

configuration

management  mechanism

generation  GC  effects

summarized  rules job  history

   How  to  do  it?  dataflowàobjectsàusage   �

n  Rules-­‐statistics  based  approach  

n  RULE  1.  𝑀𝑎𝑝𝑂𝑈≈𝑓(𝑐𝑜𝑛𝑓)  n  RULE  2.  𝑀𝑎𝑝𝑁𝐺𝑈≈𝑓(𝑋𝑚𝑥,𝑅𝑒𝑐𝑜𝑟𝑑𝑠,  𝑇𝑒𝑚𝑝𝑂𝑏𝑗𝑠)  n  RULE  3.  𝑅𝑒𝑑𝑁𝐺𝑈≈𝑓(𝑋𝑚𝑠,𝑋𝑚𝑥,  𝑅𝑒𝑐𝑜𝑟𝑑𝑠,  𝑇𝑒𝑚𝑝𝑂𝑏𝑗𝑠)  

n  RULE  4.  𝑅𝑒𝑑𝑂𝑈≈𝑓(𝑋𝑚𝑠,  𝑋𝑚𝑥,  𝑅𝑒𝑐𝑜𝑟𝑑𝑠,  𝑇𝑒𝑚𝑝𝑂𝑏𝑗𝑠)

Size  of  in-­‐memory  objects  

Management  mechanism  

GC    effects  

rules  of  memory  usage  

Mapper  JVMMapper  JVM

NewOld

Eden

S1

S0Old

InMemSeg

InMemSeg

InMemSeg

RecordsTmpObjs

New

Eden

S1

S0

RecordsTmpObjs

partTmpObjs

SpillBuffer

part  TmpObjs

Reducer  JVMReducer  JVM

   How  to  do  it?  dataflowàobjectsàusage   �map()          {      }  reduce()  {      }  Pig  /HiveQL  

Data  Configura<on  

User  layer

Framework    layer

Execution  layer

Memory

JVM

disk Dataflow  volume  

In-­‐memory  objects’  size  

rules  of  memory  usage  

memory  usage  

       Evaluation �

𝑟𝑒𝑙𝑎𝑡𝑖𝑣𝑒  𝑒𝑟𝑟𝑜𝑟=  |𝑒𝑚𝑢  −𝑟𝑚𝑢/𝑟𝑚𝑢 |  ∗100%  

Buffer  size Reducer  number

Max-­‐min  heap  size Split  size

       How  about  the  related  work? �

n  Execution  time  estimation  •  ParaTimer,  KAMD,  ARIA  and  etc.  

n  Find  optimum  configurations  •  Starfish  can  predict  job’s  performance  (mainly  

runtime)  with  different  configurations  and  has  a  cost-­‐based  optimizer  to  find  the  optimum  configuration.  

n  Find  optimum  GC  policy  for  multicore  MapReduce  •  Garbage  collection  auto-­‐tuning  for  java  MapReduce  on  

multi-­‐cores,  ISMM  2011  

     Conclusion  and  discussion �

n  Provide  a  detailed  analysis  of  job’s  memory  usage  •  considering  dataflow  and  memory  management  from  

user-­‐level  to  JVM  internals.  

n  Develop  a  fine-­‐grained  memory  estimator  •  predict  job’s  memory  usage  in  a  large  space  of  

configurations.  

n  FMEM  will  be  improved  to  do  auto-­‐configuration,  auto-­‐optimization  and  etc.  •  JIRA  MAPREDUCE-­‐4882  &  MAPREDUCE-­‐4883

Thanks,  Q&A �