deterministic container resource management in derivative clouds · 2020-04-17 · deterministic...

Deterministic Container Resource Management inDerivative Clouds

Chandra Prakash, Prashanth, Umesh Bellur, Purushottam Kulkarni

Department of Computer Science and EngineeringIndian Institute of Technology Bombay

19th April, 2018

International Conference on Cloud Engineering

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 1 / 26

Derivative cloud

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 2 / 26

Dual control over resources

I Hypervisor and guest OS both control the same resourcesI Hypervisor not aware of containers requirements

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 3 / 26

Hypervisor mechanism under consideration

I Ballooning is used to achieve memory overcommitment1

I vCPU scaling is used to reduce scheduling overheads in overcommitted situation 2

1C. A. Waldspurger, "Memory resource management in vmware esx server," ACM SIGOPS Operating Systems Review, vol.

36, no. SI, pp. 181-194, 2002.2

L. Cheng, J. Rao, and F. Lau, "vscale: automatic and efficient processor scaling for smp virtual machines," in Proceedingsof the 11th European Conference on Computer Systems, ACM, 2016.

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 4 / 26

Undesirable effects due to ballooning

Happens Desired

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 5 / 26

Existing memory reclamation in containers

I Memory provisioning knobs: Hard-Limit and Soft-LimitI exceed: difference between memory usage and Soft-LimitI SMR (Soft Memory Reclaimed): memory reclaimed from local LRUI GLR (Global LRU Reclaimed): memory reclaimed from global LRU

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 6 / 26

Existing memory reclamation in containers

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 7 / 26

Impact of ballooning

Set-up

I Memory reclamation rate: 2 GB every 30 seconds (generatedfrom host after 100 seconds)

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 8 / 26

Impact of ballooning

Default configuration of four containersContainer Hard-limit(GB) Soft-limit(GB) Key size (# records)Redis-Low 2 0.5 500KRedis-High 4 1 1000KMongo-Low 2 0.5 500KMongo-High 4 1 1000K

0

0.5

1

1.5

2

2.5

3

3.5

0 50 100 150 200 250 300 350 400

Memory usage ratio

Time (sec)

MongoDB ratio

Redis ratio

Desired ratio

Memory usage ratio

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 9 / 26

Impact of ballooning

I Existing knobs (limits) do not guarantee proportionatememory allocation during memory pressure situations

0

0.5

1

1.5

2

2.5

3

3.5

0 50 100 150 200 250 300 350 400

Memory usage ratio

Time (sec)

MongoDB ratio

Redis ratio

Desired ratio

Memory usage ratio

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 10 / 26

CPU provisioning Issues

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 11 / 26

Impact of vCPU scaling

Experimental setupVM configuration 7 vCPUs and 8GB Memory

Number of containers inside VM 3

CPU allocation ratio 1:1:4

Benchmark Sysbench

vCPU scaling down frequency 1 vCPU every 120s (vCPU1,2,3,&4)

vCPU mapping using cpuset.cpus C1: vCPU1, C2: vCPU2, C3: vCPU3,4,5,&6

0

100

200

300

400

1-120 121-240 241-360 361-480 481-600

Avg. %CPU utilization

Time Range(S)

Container1

Container2

Container3

CPU utilization without pinning

0

100

200

300

400

1-120 121-240 241-360 361-480 481-600

Avg. %CPU utilization

Time Range(S)

Container1

Container2

Container3

CPU utilization with pinning

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 12 / 26

Impact of vCPU scaling

I Pinning and scaling =⇒ non-deterministic CPU utilizationI Desired goal: achieve pinning benefits + maintain CPU share

0

100

200

300

400

1-120 121-240 241-360 361-480 481-600

Avg. %CPU utilization

Time Range(S)

Container1

Container2

Container3

CPU utilization without pinning

0

100

200

300

400

1-120 121-240 241-360 361-480 481-600

Avg. %CPU utilization

Time Range(S)

Container1

Container2

Container3

CPU utilization with pinning

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 13 / 26

Summary of issues in nesting setup

I Ballooning may fail to satisfy container requirements

I vCPU scaling may not respect cpu share with cpu pinning

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 14 / 26

Our approach

I Native cloud provider can be public or privateI We can’t control or change hypervisor in case of public cloudI We provide solution at guest OS level

Default approach Our approach

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 15 / 26

Proposed memory policies

Proportionate memory allocationI Allocate memory according to credit share of containers

Application-specific differentiated memory reclamationI Protect memory sensitive container(s) from memory reclamation

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 16 / 26

Proposed CPU policies

Maximize dedicated vCPU while maintaining allocation ratioI To get maximum benefits of pinning

Provide pinned vCPU(s) to a subset of containersI Based on application nature or user requirement

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 17 / 26

Contribution

I Modified the memory reclamation logic in memory cgroupsubsystem

I Provided an additional definition of exceed

I Performed several modifications in Linux kernel» Added extra parameters in memory and cpu cgroups» Added control to maximize SMR» Provided knob to control reclamation chunk size

I Created a cpuset calculator in user space

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 18 / 26

Modified memory reclamation

» exceed = memory_usage – proportionate_share

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 19 / 26

Effectiveness of memory policies

I Ratio of memory weights: 1:2

0

0.5

1

1.5

2

2.5

3

3.5

0 50 100 150 200 250 300 350 400

MongoDB ratioRedis ratio

Desired ratio

Time (sec)

Mem

ory

usa

ge r

atio

pressureNo memory

Memory pressure exceed of all ≰ 0

Memory pressure exceed of all ≤ 0

Memory usage ratio (default)

0

0.5

1

1.5

2

2.5

3

0 50 100 150 200 250 300 350 400

MongoDB ratioRedis ratio

Desired ratio

Time (sec)M

emo

ry u

sage

rat

io

Memory usage ratio (with control)

I Able to maintain memory usage ratio when exceed of allcontainers become less than or equal zero (after 300 second)

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 20 / 26

Application specific reclamation

0

200

400

600

800

1000

1200

1400

0 100 200 300 400 500 600

Memory usage (MB)

Time (sec)

TPC-C: default

TPC-C: with control

YCSB: default

YCSB: with control

Free Memory

Memory usage

0

200

400

600

800

1000

1200

1400

1600

1800

0 100 200 300 400 500 600

Throughput (op/sec)

Time (sec)

YCSB: default

YCSB: with control

YCSB throughput

I Memory is not reclaimed from YCSB application container(memory sensitive) and it’s throughput remains intact

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 21 / 26

vCPU reallocation design

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 22 / 26

Effectiveness of vCPU reallocationExperimental setup

VM configuration 7 vCPUs and 8GB Memory

Number of containers inside VM 3

CPU allocation ratio 1:4:5

Benchmark Sysbench & Twitter

VCPU scaling down frequency 1 vCPU every 120s (vCPU1,2,3,& 4)

0

100

200

300

400

1-120 121-240 241-360 361-480 481-600

Avg. %CPU utilization

Time Range(S)

Container1

Container2

Container3

CPU utilization by each container

0

500

1000

1500

2000

2500

3000

3500

0 100 200 300 400 500 600Throughput (op/sec)

Time (sec)

With pinningWithout pinning

Twitter throughput with scaling down vCPUs

I Able to maintain CPU share along with pinning

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 23 / 26

Conclusion & future work

Conclusion:I Quantified the impact of hypervisor actions on containers running

inside VMI Proposed user-defined policies to mitigate the impact of

hypervisor actionsI Demonstrated the effectiveness of memory and CPU policies

empiricallyFuture work:

I Design an efficient algorithm for container placement in derivative(nested) setup

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 24 / 26

Thank youQuestions???

Email id: chandrap@cse.iitb.ac.in

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 25 / 26

Modifications in Linux kernel

I Added a weight parameter in memory cgroup and a pinparameter in cpu cgroup.

I Modified the balance_pgdat() routine (Linux kernel version 4.7).Listing 1 : Original reclamation logic

For every rec lamat ion request :SMR( ) ;GLR ( ) ;

Listing 2 : Modified reclamation logic

For every rec lamat ion request :NoOfReclaimedPages = SMR( ) ;

i f ( NoOfReclaimedPages ==0) :GLR ( ) ;

I Created a kernel module to control the reclamation chunk size.

Chandra Prakash Deterministic Container Resource Management in Derivative Clouds 26 / 26