Enabling Dynamic Voltage and Frequency Scaling in

Multicore Architectures

by

Amithash Prasad

B.S., Visveswaraya Technical University, 2005

A thesis submitted to the

Faculty of the Graduate School of the

University of Colorado in partial fulfillment

of the requirements for the degree of

Master of Science

Department of Electrical and Computer Engineering

2009

This thesis entitled:Enabling Dynamic Voltage and Frequency Scaling in Multicore Architectures

written by Amithash Prasadhas been approved for the Department of Electrical and Computer Engineering

Prof. Dan Connors (chair)

Prof. Manish Vachharajani

Date

The final copy of this thesis has been examined by the signatories, and we find that both thecontent and the form meet acceptable presentation standards of scholarly work in the above

mentioned discipline.

iii

Prasad, Amithash (M.S., Electrical Engineering)

Enabling Dynamic Voltage and Frequency Scaling in Multicore Architectures

Thesis directed by Prof. Dan Connors (chair)

Traditional operating system methodologies in controlling the voltage and frequency

configuration of the machine are mostly based on ad-hoc means, thermal emergencies or con-

straining the power consumption of the system. Issues such as low memory latency, cache shar-

ing, depleting memory bandwidth have for long hindered clock speed scaling. Higher clock

speed no longer indicate better performance.

This thesis presents a scheduling methodology for multicore processors which maps run-

ning applications to cores executing at varied clock speeds based on their runtime performance

characteristic. Two schemes of mapping tasks to cores were designed. An asynchronously run

power optimizer was devised to adapt to the needs of the current workload in terms of core

clock speed on a multicore system accomplished by utilizing the information provided by the

scheduler about the needs of the current workload. The entire system was implemented as a

module for the Linux kernel.

In addition to these contributions, this thesis performs an extensive analysis on the system

for six selected workloads to analyze the effects on performance, power and energy efficiency.

Additionally this thesis presents future possible applications of the developed infrastructure

to aid compiler and user-space runtime designers to utilize the framework and provide useful

information about task characteristic and phase behavior to the operating system scheduler for

better task to clock speed assignments.

Dedication

This thesis is dedicated to my fiance Sushma for listening to me whine during the initial part of

this thesis when whatever I touched seemed to crash; For listening to me gloat in pride when

things worked for a change, and finally for proof reading my thesis when I realized that I have

forgotten the entire English language.

v

Acknowledgements

There is a saying in Sanskrit guru devo bavaha roughly translating to A professor is

equivalent to god. Thus I would like to begin my acknowledgement by thanking those who

have imparted me the most important wealth: knowledge.

First, I would like to thank my advisers Dan Connors and Manish Vachharajani for their

advice, tutoring, and incredible patience. I am grateful for the courses they taught which got

me interested in computer engineering and lured me away from control systems which was my

initial idea of a master’s degree.

Second, I’d like to thank Professor Dirk Grunwald for his motivation and suggestions

every time I stumbled into his office without an appointment which shows the level of his

patience and the willingness to help.

Third, I’d like to thank Tipp Moseley a fellow graduate student for helping me out during

my initial introduction into kernel hacking, and answering my questions whenever I felt that I

was somehow within the grasp of a black hole. This does not exclude the very helpful and

friendly Linux kernel community who helped a novice transition into a real kernel hacker. This

would not end without thanking Linus Torvalds for creating the Linux kernel in the first place.

Finally, I’d like to thank my parents for motivating me all along, understanding when I

barely call them and to have faith in me and all my endeavors (no matter how foolish it might

sound).

vi

Contents

Chapter

1 Introduction 1

2 Impact of clock speed scaling on application performance 4

2.1 Performance behavior of SPEC workloads . . . . . . . . . . . . . . . . . . . . 4

2.2 Quantifying performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Performance directed scheduling on multicore processors 10

3.1 Hierarchical processor organization . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Hardware performance counters . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.3 Scheduling methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.3.1 Performance state estimation with the Ladder approach (LEA) . . . . . 14

3.3.2 Performance state estimation with the select approach (SEA) . . . . . . 15

3.4 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.5 Static/Fixed layout of processor cores . . . . . . . . . . . . . . . . . . . . . . 17

4 Power-aware throughput management on multicore systems 19

4.1 Common power management systems . . . . . . . . . . . . . . . . . . . . . . 20

4.2 Overview of the Linux cpufreq architecture . . . . . . . . . . . . . . . . . . . 21

4.3 Proposed system overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.4 Problem definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

vii

4.5 The delta constrained mutation algorithm . . . . . . . . . . . . . . . . . . . . 27

4.5.1 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.5.2 The Manhattan matrix . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.5.3 The Manhattan weight vector . . . . . . . . . . . . . . . . . . . . . . 34

4.5.4 Cooperative demand distribution: The demand field . . . . . . . . . . . 34

4.5.5 Greedy performance state selection . . . . . . . . . . . . . . . . . . . 35

4.5.6 Greedy processor selection and transition . . . . . . . . . . . . . . . . 36

4.5.7 Parameter change and termination conditions . . . . . . . . . . . . . . 36

5 Experimental setup and results 38

5.1 Trends along delta and interval . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.2 Trends along maximum allowed mutation rate . . . . . . . . . . . . . . . . . . 43

5.3 Comparing various methodologies and varying workloads . . . . . . . . . . . . 45

6 Future work 50

7 Summary and Conclusion 51

Bibliography 53

Appendix

A AMD Opteron capabilities 56

B Mutation time-line per workload 57

C Source code 61

viii

Tables

Table

2.1 Classification of the SPEC 2006 benchmarks . . . . . . . . . . . . . . . . . . 6

2.2 Workload members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.1 Threshold values for the select evaluation . . . . . . . . . . . . . . . . . . . . 16

3.2 Experimental layouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

A.1 Correlation between performance states with clock speed and power consumption 56

ix

Figures

Figure

2.1 IPC classification and speedup trends for the SPEC 2006 benchmarks . . . . . 5

2.2 Speedup achieved by characterized workloads . . . . . . . . . . . . . . . . . . 7

2.3 Speedup dependence with IPC . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Energy consumption per instruction variance with IPC . . . . . . . . . . . . . 9

3.1 Multi-processor organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.2 Hardware performance monitoring counters . . . . . . . . . . . . . . . . . . . 12

3.3 Scheduling state diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4 Performance directed migration . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.5 Performance state estimation with the Ladder approach (LEA) . . . . . . . . . 15

3.6 Performance state estimation with the select approach (SEA) . . . . . . . . . . 16

3.7 Slowdown vs power consumption for fixed layouts . . . . . . . . . . . . . . . 18

4.1 The Seeker governor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.2 Hypothetical time line displaying the invocation frequencies of the scheduler

and the mutator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.3 Example mutation with the Manhattan distance between L (layout before mu-

tation) and L′ (layout after mutation) equal to 3 . . . . . . . . . . . . . . . . . 25

4.4 Mutation algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.5 State transition diagram for processor i currently in performance state Li . . . . 31

x

5.1 The Seeker infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

5.2 Variation of average slowdown with delta and interval for the ladder (a) and the

select (b) estimation approach . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.3 Variation of average power savings with delta and interval for the ladder (a) and

the select (b) estimation approach . . . . . . . . . . . . . . . . . . . . . . . . 43

5.4 Variation of average energy efficiency (EPI) improvement with delta and inter-

val for the ladder (a) and the select (b) estimation approach . . . . . . . . . . . 44

5.5 Variation of average power savings, slowdown and energy efficiency improve-

ment with maximum mutation rate for the LEA (a) and the SEA (b) scheduling

systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.6 Variation of power savings, energy efficiency improvement and slowdown with

each workload for the ladder estimation approach (LEA) with the delta mutation

engine at ∆ = 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.7 Variation of power savings, energy efficiency improvement and slowdown with

each workload for the select estimation approach (SEA) with the delta mutation

engine at ∆ = 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.8 Variation of power savings, energy efficiency improvement and slowdown with

each workload for the Ondemand governor . . . . . . . . . . . . . . . . . . . 49

B.1 Adaptation time-line for the High workload with the ladder (a) and select (a)

scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

B.2 Adaptation time-line for the Low workload with the ladder (a) and select (a)

scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

B.3 Adaptation time-line for the Low-High workload with the ladder (a) and select

(a) scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

B.4 Adaptation time-line for the PHigh-PLow workload with the ladder (a) and se-

lect (a) scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

xi

B.5 Adaptation time-line for the PLow-Low workload with the ladder (a) and select

(a) scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

B.6 Adaptation time-line for the PLow-High workload with the ladder (a) and select

(a) scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Chapter 1

Introduction

Emerging technology constraints are slowing the rate of performance growth in com-

puter systems. Specifically, designers are finding difficulties addressing strict processor power

consumption and cooling constraints. Design modifications to address power consumption gen-

erally limit processor performance and reduce peak operating frequency, thus changing the

trend of providing increased system performance every new processor generation. As such,

modern architectures have diverged from the clock speed race into the multicore era with mul-

tiple processing cores on a single chip. While the multicore design strategy improves chip

power density, there remains a significant potential in improving run-time power utilization by

dynamically changing per-core clock frequency and voltage.

Power consumption is directly proportional to the clock speed (frequency) and the square

of the operating voltage of the processor (Power ∝ frequency × V oltage2). In order to con-

serve energy, designers have explored methods to manipulate these parameters using circuit

design techniques and runtime software driven techniques. Implementations of DVFS (Dy-

namic voltage and frequency scaling) schemes divides the processor operation to performance

states or P-states with fixed operating voltage and clock speeds. Architectures supporting mul-

tiple such P-states require active runtime support to fully explore the performance and power

saving potential of the DVFS approach.

The schemes for manipulating DVFS configurations of a processor can be subdivided

2

into three broad categories: Hardware approach [1], [2] and [3] traditionally propose monitor-

ing hardware to predict execution patterns and manipulate the system’s DVFS configuration.

Hardware methods are immutable implementations in silicon which cannot be changed based

on design and policy variations and hence avoided by many chip manufacturers. The second

category is the management of power, controlled by compiler and user space runtime systems

which expect such capabilities to be exported by the operating system as discussed in [4], [5], [6]

and trace driven methods for power management as discussed in [7] and [8]. Operating system

techniques [9], [10], [11] and [12] generally use some of runtime performance monitoring ca-

pabilities exported from the hardware in order to assist in detecting runtime phases, and include

software policies to ultimately make decision for power management.

Majority of DVFS implementations at the operating system level are load based where

the system’s DVFS configuration is varied based upon extremely coarse-grain levels of proces-

sor activity. Even though effective in conserving power such approaches are largely unaware of

the workload currently executing on the system. Load based decisions are best left to technolo-

gies which decide whether the processing element is active or asleep (state at which the power

consumption is minimal) and is studied in [11] which predicts usage patterns of various devices

on the system.

This work proposes techniques which tie the scheduler and the system power optimizer

to constrain the number of DVFS level changes both over time and intensity with the goal of

energy savings without harming individual performance and throughput. Limiting DVFS level

transitions is important to limit instabilities endured by the system due to rapid DVFS transitions

as explored by [13].

This thesis makes the following contributions to this area of study:

(1) Method of scheduling to adjust DVFS with strict constraints on limiting the amount of

performance lost compared to full frequency mode. The system provides significant

power savings of up to 40% with a performance loss of up to 20% with a median

3

performance loss at around 10-15%.

(2) Method to schedule tasks based on their performance requirement on a multi processor

environment with varying per-core clock speeds in order to maximize power savings

while minimizing performance lost.

(3) Investigation of the impact of DVFS level changes per second on the ability to improve

power savings or performance impact.

(4) Design of a multi-task scheduling model that accounts for joule-per-instruction work

metric.

This thesis is organized as follows: Chapter 2 discusses the performance variation of real

world applications with increases in clock speed. Chapter 3 introduces the performance directed

scheduler. Chapter 4 proposes a methodology to mutate voltage and frequency configuration of

a system based on the workload demands. Chapter 5 describes the experimental procedure and

results obtained. Chapter 6 comments on the future work in this area and finally chapter 7

concludes this thesis.

Chapter 2

Impact of clock speed scaling on application performance

Throughput based computing has long been solely dependent on clock speed advantages

for faster application execution. With the onset of multicore processors, a large portion of

research has been devoted to parallelization of sequential applications. The performance of most

real world applications are either dependent on memory speeds or I/O (Input/Output) latency.

Parallel executing threads on a multicore system eventually saturate memory or I/O devices and

thus obtain no advantage of clock speed scaling. This chapter explores the impact of scaling

clock speeds on application performance and energy efficiency in multicore processors.

2.1 Performance behavior of SPEC workloads

In order to study the behavior of real applications on different voltage and frequency

settings, fourteen of the SPEC2006 [14] benchmark suite were run with varied clock speed from

1100 MHz to 2200 MHz on an AMD Opteron quad-core processor and were plotted showing

the percentage duration spent at an IPC (Instruction per Cycle) higher than 1.0 and the speed-up

in terms of execution time achieved from that when run with the lowest clock speed (1100 MHz)

shown in Figure 2.1. The X-axis of this graph is the percentage of time the benchmark spent

running at an IPC level greater than 1.0, the Y-axis is the clock speed at which the benchmark

was run at, and finally the Z-axis is the Speed-up achieved. It can be observed that the return

5

on investment (Higher clock speed) is low for tasks with a low percentage of runtime at an IPC

greater than 1.0.

Figure 2.1: IPC classification and speedup trends for the SPEC 2006 benchmarks

The SPEC2006 [14] benchmark suite is a set of real-world applications designed to run

with minimal I/O and hence best used to characterize and study the CPU (central processing

unit) and memory subsystem in modern processors. The applications are typical workloads

used in the field of throughput based computation and is widely used test or design processor

systems. Based on this observation the benchmarks were categorized into four categories: Low,

High, Phased Low and Phased High and shown in Table 2.1.

Table 2.2 shows the breakdown of six workloads (Low, High, Low-High PHigh-PLow,

PLow-High and PLow-Low) comprising of 4 benchmarks each were created (shown in Ta-

ble 2.2) based on their IPC classification in Table 2.1. It must be noted that even though these

workloads were grouped based on their IPC characteristics, it is not uncommon to have work-

loads in the real world comprising of subsets of each benchmarks. The six set of workloads

6Category BenchmarksLow (90%+executes belowand equal toIPC=1.0)

mcf milc libquantum lbm omnetpp

High (90%+executes aboveIPC=1.0)

bzip2 povray hmmer sjeng h264ref

Phased Low(50%+ executesabove IPC=1.0)

gobmk sphinx3

Phased High(50%+ executesbelow and equalto IPC=1.0)

astar xalancbmk

Table 2.1: Classification of the SPEC 2006 benchmarks

provide a fair but experimentally feasible set of applications to estimate the impact of architec-

ture or operating system design.

Workload BenchmarksLow milc libquantum lbm omnetppHigh bzip2 povray hmmer sjengLow-High bzip2 povray milc omnetppPHigh-PLow gobmk sphinx3 astar xalancbmkPLow-High astar xalancbmk povray hmmerPLow-Low astar xalancbmk milc omnetpp

Table 2.2: Workload members

Figure 2.2 shows that the workload High category shows the greatest improvement with

frequency, achieving a nearly 2x improvement using the highest frequency for each core in the

quad-core system. Otherwise, the other categories of workloads appear to benefit from higher

frequency, but are bounded to 1.5x of the baseline execution time. The Low workload is ob-

served to have a very high variance in terms of speedup and shows that clock speeds above 1700

MHz has no effect on the speedup achieved. This motivates the construction of a performance

directed scheduler aware of the IPC level at which tasks are executing and described in further

7

sections.

Figure 2.2: Speedup achieved by characterized workloads

2.2 Quantifying performance

An adaptive system needs a quantitative measure of performance. Section 2.1 indicated

that IPC can be used as such a measure. In order to study the throughput and energy consump-

tion behavior of workloads, a hardware monitoring-based framework was constructed to col-

lect and correlate per-application IPC data. Figure 2.3 was constructed displaying the speedup

which was achieved in terms of throughput when compared to the same sample executed at the

base clock speed of 1100 MHz. The analysis provides an aid in observing the exact IPC values

at which a higher clock speed is no better in increasing the throughput. Figure 2.3 shows that be-

8

yond IPC = 1.0 (box High IPC), a higher clock speed will always provide better throughput to

the system as demonstrated by the separation between curves corresponding to each frequency.

Samples with IPC in between 0.5 and 1.0 (box Moderate IPC) shows that clock speeds greater

than 2000MHz have no significant effect in improving the throughput demonstrated by an in-

significant separation between the curves corresponding to 2000MHz and 22000MHz. Finally

IPC lower than 0.5 (box Low IPC), indicate that clock speed increases not not dramatically

improve the throughput of the system (There are not clearly defined separation between the

curves). This motivates a system to manage DVFS configuration by detecting the IPC phase

of applications to conserve power by scaling the clock speed down during phases equivalent to

Box Low IPC.

Figure 2.3: Speedup dependence with IPC

Energy consumption and hence monetary cost of maintaining a server system during

application execution (Including cooling costs) is directly proportional to the product of the

9

operational power consumption and the execution time measured in Joules. Energy utilization

efficiency of a processor can be measured as the energy consumed per executed instruction or

EPI (Joules / Instruction). The data obtained by the above experiment aided in computing the

energy consumption of each sample by multiplying the power (Obtained by correlation provided

in Appendix A) with the execution time of each sample. Finally the EPI value of the sample was

derived by dividing the energy consumption by the total number of instructions in each sample.

Figure 2.4 demonstrates that executing applications with low IPC at higher clock speeds results

in poor energy utilization. EPI exponentially falls at higher values of IPC and making the

energy utilization consistent across all clock speeds. This should further motivate the use of

DVFS techniques to improve energy utilization efficiency for throughput based systems.

Figure 2.4: Energy consumption per instruction variance with IPC

Chapter 3

Performance directed scheduling on multicore processors

Current implementations of schedulers in modern operating systems recognize the en-

vironment as homogeneous in nature while technologies such as DVFS introduce an inherent

asymmetry in a multicore system. Considerable exploration in terms of asymmetric scheduling

support has been investigated in [15] and [16]. However, the work ignores the fact that certain

workloads do not gain sufficient advantages from increased clock speeds discovered in Chap-

ter 2 which could assist in conserving power by scaling the clock speed down during phases of

low IPC.

In a multicore environment, there are many facets for performance loss with the most

common reasons being high IO and memory latencies which is further aggravated by cache

sharing. A novel statistical learning method to predict voltage and frequency requirements by

tasks was proposed in [5], but the method required rigorous statistical learning making it time

consuming and impossible to implement at the context of a scheduler. For example even with a

quad core system and a scheduling frequency of 1000Hz, there is a possibility of 1000 × 4 =

4000 performance state transitions every second which can cause instabilities as shown in [13].

A simpler method was proposed in [10] and [2], but comes at the cost of possible rapid voltage

and frequency transitions which may lead to instabilities in emergencies

11

3.1 Hierarchical processor organization

Chapter 2 provide a strong motivation to scale voltage and frequency of processors based

on the performance of the workload. While most of the research pertaining to workload based

DVFS direct their attention of changing the voltage and frequency levels based on the task’s

demand, none consider the possibility that such a performance state might already be available

on another core which can be remedied by a simple migration without the causality of rapid

performance state transitions. The motivation to substitute migration for frequency transition

is further enhanced with the current multicore race where the probability of such a situation

improves proportionally.

In order to enable scheduling on asymmetric (cores executing with varied clock speeds)

multicore processors, the first implementation step was to organize the processor set as groups

with equal performance states. The term performance state or P-State will be used to differ-

entiate varied voltage and frequency configurations that a processor is allowed to have, and is

indicated by P0 through Pm−1 where, P0 is the configuration with the lowest clock speed and

voltage (lower power consumption state). Each set maintains the usage in terms of actively

running tasks in the particular group to enable load balancing between groups of varied perfor-

mance states. This view of the processor set is shown in Figure 3.1 and mathematically viewed

as a vector of sets shown in Equation (3.1) where, CPi is the set of processors which are at a

performance state of Pi.

Ls =[CP0 CP1 ... CPm−1

](3.1)

3.2 Hardware performance counters

In order to quantitatively measure the performance of each task, a driver was developed

12

Figure 3.1: Multi-processor organization

to read and configure the hardware performance counters present in the AMD and Intel Archi-

tectures. Two counters were configured to measure the sample IPC of an executing workload as

shown in Figure 3.2. The first counter tracked the total number of executed instructions while

the second tracked the total number of real clocks. It must be noted that the real clock rate (real

clocks per second) is the clock speed of the current DVFS configuration.

Every request for IPC made to the performance monitoring subsystem, causes both these

performance counters to be read based on which the IPC (instructions per clock) is computed as

a fixed point value of precision 0.125. After each successful transaction, the retired instruction

and real clock counters are cleared and continue incrementing through the next execution sam-

ple. The performance monitoring subsystem is unaware of the current executing task and it is

the scheduler’s responsibility to read the IPC level before switching out tasks from the running

state.

Figure 3.2: Hardware performance monitoring counters

13

3.3 Scheduling methodology

Most time sharing scheduling systems regularly switch out a running task to another in

order to provide fair equal execution times for all tasks. This execution slice is commonly re-

ferred to as the scheduling quanta. The Linux scheduling system maintains separate run-queues

for each processing element done in order to reduce the scheduling algorithmic complexity

to O(1). Migration, a procedure of moving a runnable task from one processor to another is

reduced to moving the task from one run-queue to another.

Figure 3.3 shows the state diagram of the performance directed scheduler which is a mod-

ification of the default Linux scheduler with an additional pseudo state (Performance directed

Migration) introduced at the transition away from the running state At this state, as shown in

Figure 3.4, the hardware performance counters are queried for the value of the current IPC (In-

structions per cycle). Based on this measure of the task’s performance, a mapping from IPC to

a required performance state (P ′) is made. This mapping can potentially be done is a number of

ways of which two, Ladder where the clock speed is determined based whether the IPC is lower

or higher to preset thresholds and Select where the clock speed is determined by comparing the

IPC to a table of IPC ranges each mapping to a specific clock speed; were implemented and

evaluated. These methods are further described in the following two subsections.

Once P ′ is estimated, the layout Ls is is queried for the total number of processors in the

setCP ′ and the total number of tasks in the ready or running state for processor inCP ′ . If the set

CP ′ is populated and the total number of tasks executing on the processors in set CP ′ is lesser

than or equal to the number of processors in CP ′ , then P ′′ = P ′. But if such is not the case,

then the layout Ls is searched for a set CP ′′ such that P ′′ is closest to P ′, and CP ′′ is populated

and the total number of tasks executing in the set CP ′′ is lesser than the number of processors

in CP ′′ . If all fails, then the layout is searched for a state P ′′ such that CP ′′ is populated and the

load (Computed as the total number of tasks divided by the number of processors) is minimum.

14

Finally the task is migrated to the set of processors in CP ′′ . Decisions on the exact processor a

task has to execute on is made by the underlying native scheduler of the operating system and

the PDS is completely oblivious to any further detail. As an interface to the mutator which will

be explained in Chapter 4, PDS increments the cell corresponding to P ′ in the demand vector

D (DP ′ = DP ′ + 1).

Figure 3.3: Scheduling state diagram

Figure 3.4: Performance directed migration

3.3.1 Performance state estimation with the Ladder approach (LEA)

15

The Ladder Estimation Approach sports a simple decision procedure. If IPC > H , then

P ′ is chosen such that is is higher than the state at which the task is currently executing with

(P ′ = P + 1). If IPC < L, then P ′ is chosen to be one less than the state at which the task

is executing with (P ′ = P − 1). Otherwise P ′ is chosen to equal P . The threshold values of

H and L were chosen to equal 0.875 and 0.5 respectively by observing Figure 2.3 (Chapter 2)

to be the edges of High IPC and Low IPC ranges. The advantage of the procedure is the lower

choice (two: H and L) available to the system administrator. It is clear that the wrong choice of

these thresholds can cause undesirable power and performance behavior of the system.

Figure 3.5: Performance state estimation with the Ladder approach (LEA)

3.3.2 Performance state estimation with the select approach (SEA)

A logical extension of the LEA system, is the Select Estimation Approach that maps

clock speeds to specific ranges of IPC as shown in Figure 3.6. Such a system was implemented

with threshold values shown in Table 3.1. These values are selected by carefully observing

Figure 2.3 (Chapter 2) and approximating the values of IPC for which each clock speed saturates

(Increasing clock speed does not gain significant improvement in speedup). The advantage of

this method is in the independence from the current performance state but comes at the cost of

complexity requiring rigorous experimental evaluation for each architecture.

16

Figure 3.6: Performance state estimation with the select approach (SEA)

Threshold ValueT0 0.25T1 0.50T2 0.75T3 1.25

Table 3.1: Threshold values for the select evaluation

A final note of the difference between the LEA and SEA systems: The LEA system

chooses the next higher or lower performance state based on the threshold, while the SEA

system chooses a specific performance state. The SEA and LEA systems were evaluated with

the delta based mutation engine described in Chapter 4.

3.4 Experimental setup

The experiments were conducted on a AMD quad-core Barcelona which allows indi-

vidual processor cores to be set to different clock speeds. The set of workloads described in

Table 2.2, Chapter 2 (groups of SPEC2006 benchmarks) were started on the layouts provided

17

in Table 3.2 (increasing from a to n each possessing faster cores and in effect higher power

consumption), and the execution time of each member benchmark was measured and summed

to arrive at a single execution time per workload. The experiment was repeated three times and

the mean of the execution times were compared with that of the [4, 4, 4, 4] layout to get the

percentage slow down.

Layout Name PC0 PC1 PC2 PC3

a 0 0 0 0b 0 0 1 1c 0 0 2 2d 0 0 0 4e 1 1 2 2f 0 0 3 3g 0 0 4 4h 1 1 3 3i 1 1 4 4j 2 2 3 3k 2 2 4 4l 0 4 4 4m 3 3 4 4n 4 4 4 4

Table 3.2: Experimental layouts

3.5 Static/Fixed layout of processor cores

The results of the experiment described above are plotted with the X-axis being the aver-

age power consumption of the processor layout and the Y-axis being the percentage slowdown

observed. One obvious trend is decrease in the percentage slowdown with the increase of power

provided per core (higher performance states available) increases. The problem with fixed static

layouts are obvious from the peaks observed whenever there is an asymmetry in the layout. An

obvious example being layout l ([0, 4, 4, 4]) where processor C0 is in state P0 while the re-

maining processors C1,C2 and C3 are in P4. Going from layout k to l which are a couple of

18

watts apart in terms of power consumption, but layout l displays 40% higher slowdown than k

[2, 2, 4, 4]. This behavior has been observed to be common when there are more tasks battling

over a lower number of processors where the time spent in migrations become noticeable and

add to the slow down of the task. The Low and High workloads are the least affected by varying

layouts which demonstrates that static layouts are best when all workloads exhibit equal and

non-varying phase behavior.

Figure 3.7: Slowdown vs power consumption for fixed layouts

Chapter 4

Power-aware throughput management on multicore systems

Power and energy management are critical to high-performance and high-throughput

environments. Servers are usually limited to the amount of peak power and energy consumed

in order to reduce maintenance cost. As an example, power and energy considerations govern

server space expansion in industries. To combat these constraints, power management systems

are becoming increasingly common in the server space. Two common methodologies exist to

reduce the power consumption of a compute element. The first among which is to turn off

processors during idle time. The second is to keep the system active but lower the operating

frequency and voltage of the processor to utilize proportionally lower power.

In order to fully utilize server nodes and minimize idle time, the current trend among in-

dustries is to run multiple virtual machines on a single server rack thus making varied workloads

common place in an otherwise single purpose server usage. This element further enhances the

requirement of dynamic non-trace driven power management systems. [17] discusses a power

optimizer specialized for a virtual machine farm, but fail to recognize the workload character-

istics of these systems.

Lowering frequency has a pleasant benefit of reducing the power consumption and hence

the energy and cooling costs. But as shown in Chapter 3, Figures 2.3 and 2.4, when the average

IPC of an application is high, reduction in frequency only causes the application to execute for

a proportionally longer duration and hence having no energy benefit (Figure 2.4). The only

20

advantage of reducing the frequency is the reduction of energy supply into the system per unit

time and hence possibly lower heat dissipation.

[5], [10] and [2] propose adapting the clock speed of the processing element based on the

current application’s demand. These methodologies assume that DVFS transitions are local to

that processor core. Some multicore processors have dependency between processor cores (tran-

sitioning one might potentially transition the other) or systems with symmetric multi-threaded

features where a single processor core is visible to the operating system as multiple virtual

processors. Thus, applications executing on such processing elements are tied to each other

and any DVFS transition based on one application might potentially affect another negatively.

Chapter 3 showed that such transitions could be remedied with a simple processor migration

requiring power optimizers to react to the needs of a scheduler.

4.1 Common power management systems

The most popular among power management techniques are load directed systems which

transition a processor to higher or lower performance states based on the current load of the

system. Two of these techniques: ondemand [18] and conservative are implemented within the

thesis infrastructure. The ondemand system raises the performance state to the highest possible

level at high loads, and reduces the performance state gradually to the lowest state during lower

load thus P ′i ,

P ′i =

Pmax : Loadi ≥ 0.8

Pi − 1 : Loadi < 0.8(4.1)

the performance state of processor i after the transition based on the current performance state

Pi and decided based upon load of processor i, loadi. The characteristics of this system is

to rapidly respond to load increase while conservatively lowering the performance state at the

end of the active state. The conservative system, on the other hand, gradually changes the

21

performance state in either direction, where P ′i ,

P ′i =

Pi + 1 : Loadi ≥ 0.8

Pi − 1 : Loadi < 0.8(4.2)

the performance state of processor i, after a transition is decided based on the load of each

processor Loadi with a typical characteristic to gradually meet the system needs and preferred

for servers with typically short running workloads.

4.2 Overview of the Linux cpufreq architecture

In order to maintain architecture independence, separation of methodology and policy,

the subsystem in the Linux kernel responsible for managing the voltage and frequency con-

figuration separates the procedure into two: cpufreq-drivers (The region enclosed by Drivers:

Figure 4.1) are responsible for the actual P-State transition and register with the intermediate

cpufreq layer. cpufreq-governors are responsible for policy and registers with the cpufreq layer.

Policy attributes are:

• How often changes to DVFS configurations are made.

• The magnitude of changes in configuration settings.

Once a working configuration is initialized, the cpufreq-governor instructs the cpufreq-

driver in an indirect fashion on the required transition. As most of the experimentation men-

tioned in this text relates to the AMD Barcelona, the powernow-k8 cpufreq-driver was used to

initiate the transition.

The procedure of changing the DVFS configuration of a multicore processor system is

termed, for the remainder of this thesis, as mutation. The entity involved in making policy

decisions about the nature of the mutation (what processor transitions to what performance

state) is referred to as the mutator. The interval at which mutation decision is performed is

termed the mutation interval.

22

A cpufreq governor seeker was developed having kernel exported interfaces for the mu-

tator to request transitions. A simple registration method was introduced to allow the mutator

to be informed every time a transition is complete as shown in Figure 4.1. The asynchronous

call back mechanism allows the mutator to update its data structures even when an entity other

than itself ( For example, through the sysfs interface) requests for a performance state change.

Figure 4.1: The Seeker governor

4.3 Proposed system overview

In contrast to the load based power management system described in Section 4.1, a

performance directed power management system was implemented. [5], [10] and [2] describe

methodologies where DVFS transitions are invoked based on the performance requirement of

the executing workload of the system. Chapter 3 demonstrated a novel method of substituting

task migrations for DVFS transitions in a multi-processor system where each processor could

23

possible be at varied performance states. With a processor layout Ls being a vector of sets with

a length of m, which allowed the performance directed scheduler to schedule tasks on a set with

an algorithmic complexity of O(m).

Another possible view of the processor layout

Lm = [PC0PC1 ...PCn−1 ] (4.3)

where Lm is a vector of integers of length n, where each element Lmi is the performance state

of processor i. It is clear that both Ls and Lm are different views of the same information

and the proposed mutation scheme must take the responsibility to keep these two views of the

processor layout consistent. As Ls is optimized for the scheduler it will not be further discussed

in this chapter and any reference to processor layout or L will be with respect to the mutation

engine’s view of the processor layout: Lm. Thus the processor layout provides n options for

each workload in terms of performance state.

Chapter 3 demonstrated the short comings of a static processor in lieu of varying work-

load in terms of characteristics and number. In order to alleviate the negative aspects of a static

layout, a mutation scheme was developed allowing the processor layout to adapt on a fixed in-

terval, the mutation interval, to the demands of the workload (mutation interval >> scheduling

quanta). Figure 4.2 shows the time line interaction of the performance directed scheduler and

the mutation engine.

Every scheduling quanta the performance directed scheduler is invoked which measures

the performance (IPC) of the task and based on which the preferred performance state P ′, is

chosen which varies depending on the performance state evaluation method (ladder or select).

Then the corresponding element of the demand vector D is incremented (DP ′ = DP ′ + 1).

At the expiration of every mutation interval possibly incorporating numerous such evaluations

(mutation interval >> scheduling quanta), this demand vector D provides the mutation engine

with an understanding of the current performance requirements of the processor layout L. At the

end of the mutation algorithm, the demand vector D is cleared (∀i ∈ N, i < m : Di = 0). Thus

24

Figure 4.2: Hypothetical time line displaying the invocation frequencies of the scheduler andthe mutator

creating an illusion of a dynamic processor layout which adapts to the needs of the machine’s

workload.

Since rapid and drastic DVFS transitions can affect the reliability of the system [13],

operating system management policies are critical. The delta constrained mutation scheme was

developed providing the ability to control the rate of transitions by limiting DVFS transitions

to be performed globally and at fixed interval lengths termed the mutation interval. Secondly,

the delta mutation policy can limit the magnitude of mutation thus eliminating rapid transitions.

The PDS as described in Chapter 3 is performed at the resolution of a scheduler quanta, while

the DVFS transitions are performed at a resolution of the mutation interval and is a multiple

of the scheduling quanta. In order to have an absolute upper bound on the DVFS transitions,

a parameter called the delta constraint (∆) limits the maximum number of mutations that can

be performed at any particular instant (hence limiting the maximum mutations per second to

∆Mutation Interval ).

Delta (∆) for a system with n processors where each processor i is transitioned from

performance state Pi to P ′i can be defined as

∆ ≥n−1∑i=0

|Pi − P ′i | (4.4)

25

and can also be defined as the maximum Manhattan distance allowed between the layout vector

L before the mutation and the layout vector L′ after the mutation. Thus the example mutation

as shown in Figure 4.3 is allowed only for a system with a delta constraint ∆ ≥ 3.

Figure 4.3: Example mutation with the Manhattan distance between L (layout before mutation)and L′ (layout after mutation) equal to 3

4.4 Problem definition

An overview of the system required is as follows:

(1) The scheduler estimates the required performance state for each task and maintains the

demand, a monotonically increasing count, representing the number of times each state

was requested.

(2) Each processor can take performance states from 0 to m− 1.

(3) The total movements: the Manhattan distance from the current layout to the next layout

should always be less than or equal to the delta constraint (∆).

(4) The system should be partial to maintaining a core’s current performance state if such

a performance state is requested.

26

The problem can be expressed as a multiple choice knapsack problem with each variable

xi (corresponding to each processor) capable of m options thus xij

xij ∈ {0, 1}, i = 0, 1, 2, ..., n− 1; j = 0, 1, 2, ...,m− 1 (4.5)

is a 0-1 choice of selecting performance state j for processor i. As we can at most select one

performance state to a processor, the problem is restricted by

m−1∑j=0

xij = 1,∀i = 0, 1, ..., n− 1 (4.6)

which allows only one of the choices for all performance states j. The knapsack capacity is the

delta constraint (∆) thus bringing the constraint of the optimization problem to be

Subject to :n−1∑i=0

m−1∑j=0

|Li − j|xij ≤ ∆ (4.7)

where xij is the selection of the transition of processor i from performance state Li to j. Finally

the problem is to optimize the transitions in such a way as to optimally satisfy the demand

vector D by

maxn−1∑i=0

m−1∑j=0

Djxij (4.8)

Defining the problem this way, the memorization based dynamic programming method

described in [19] was implemented. Early experimental results showed that the algorithm was

inefficient for having no concept of transition direction, that is, the algorithm attempts to fill

the knapsack (performing enough transitions within the constraint of ∆) without distinguishing

between transitions which increase performance from transitions which decrease performance.

Another effect which is not accounted by the dynamic programming approach is to ig-

nore the fact that the algorithm will be repeated periodically and the current results will be

utilized as the problem during the next mutation interval. Thus the dynamic programming ap-

proach tries to perfect the selection even though the layout adaptation can always be perfected

in future mutation intervals, Implying that the required nature of adaptation is to at least gravi-

tate towards the optimal layout. This motivated the development of an iterative direction based

greedy algorithm described below.

27

4.5 The delta constrained mutation algorithm

Figure 4.4 describes the steps involved in the proposed iterative greedy algorithm. Each

sub step is described in further sections. The initialization procedure is described in Sec-

tion 4.5.1. Based on the iteration delta constraint δ, a matrix, Manhattan matrix (S) is con-

structed and described in Section 4.5.2. The Manhattan weight vector W is described in Sec-

tion 4.5.3. The cooperative demand transformation is described in 4.5.4. The greedy winning

state w selection is described in Section 4.5.5. The selection of the processor c to transition

to the winning state w is described in Section 4.5.6. Finally, the parameter adjustments and

iteration termination conditions are described in Section 4.5.7.

Figure 4.4: Mutation algorithm

28

4.5.1 Initialization

In order to effectively provide only the needed number of processors, the mutator queries

the operating system for the total number of tasks in the ready state T . The Load of the system

Load =

T : T ≤ n

n : T > n

(4.9)

in terms of number of processors required is computed where T is the total number of tasks

in the ready state in a multicore system with n on-line cores. Using an estimated load based

on idle times of processors was found to be unfruitful as it can be non-representative of the

actual computing capacity demanded by the number of active tasks. This can be aggravated in

situations where multiple tasks could be queued on a single processor.

4.5.1.1 Processor demand

The Performance Directed Scheduler as described in Chapter 3 maintains the demand

for each state in the demand vector D. The contents of this vector cannot be directly consumed

as their values pose no direct description of the demand of individual performance states. As a

direct consequence, vector demand is computed based on the projected load of the system and

the demand vector D,

demandi =Di × Load

m−1∑j=0

Dj

(4.10)

where Di is the total number of times performance state Pi was requested by the performance

directed scheduler in the previous mutation interval while the total load of the system was

estimated to be load. The vector demand is also referred to as the core demand as each element

demandi is the total number of cores demanded by the performance directed scheduler at state

Pi in order to optimally schedule the current workload.

29

4.5.1.2 Present and required power characteristic of the system

powerP the present or current power number of the multicore system,

powerP =n−1∑i=0

Li (4.11)

where n is the total number of on-line cores and L is the current layout of the system. Thus

powerP is an integer value proportional to the total power drawn by the multicore system.

powerR, the required power number

powerR =m−1∑i=0

i× demandi (4.12)

is computed based on the processor demand demand as shown below in Equation (4.12) where,

each core is capable of m performance states and demandi being the total number of cores

demanded by performance state Pi. Thus powerR is proportional to the power drawn by the

multicore system when the cores are transitioned in such a way as to satisfy the performance

directed scheduler to optimally schedule it’s current workload.

4.5.1.3 Transition direction

Based on the parameters powerP and powerR, the transition direction, trans, a tri-state

value trans ∈ {−1, 0,+1} describing the nature of mutation required

trans =

1 : powerR > powerP

0 : powerR = powerP

−1 : powerR < powerP

(4.13)

is computed. This is done by the mutation engine to evaluate the required direction in which

DVFS configuration must be made. A transition direction of trans = +1 implies that the

scheduler requires higher performance from the multicore system, while a transition direction

of trans = −1 implies the exact opposite; an indication to conserve power by reducing the

30

performance states of the multicore processor. A transition direction equal to zero implies

stability and mutations must be avoided if possible.

4.5.1.4 Poison vector and iteration delta constraint initialization

Before starting the iterations, the poison vector X of length n

∀i ∈ N, i < n;Xi = 1 (4.14)

is initialized to all ones, where each element is a binary representation (Xi ∈ {0, 1}) of a

unique processor’s current assignment states. A value Xi = 0 indicates that processor i has

been allocated a performance state and must not be re-transitioned. Finally, the iteration delta

constraint δ is initialized to ∆ (δ = ∆).

4.5.2 The Manhattan matrix

The Manhattan matrix S with n rows (one for each core) and m columns (one for each

performance state), can be viewed as a probability matrix where each element Sij indicates the

probability that processor i can be transitioned to performance state j for a system with n cores

each capable of m performance states and depicted in Figure 4.5. Thus S23 describes the weight

of processor C2 transitioning to the performance state P3. The conception of this matrix was

achieved by viewing a probability density function for each processor and was soon modified

due to the lack of floating point operations in kernel space (Even though possible, the kernel

address space does not save the floating point context and makes it’s use dangerous as the kernel

is fully preempt-able).

31

Figure 4.5: State transition diagram for processor i currently in performance state Li

The Manhattan matrix (n×m) is computed as

Sij =

0 : j < (Li − δ)

m2 × (trans2 − trans + 2) + j − Li : (Li − δ) ≤ j < Li

2m : j = Li

m2 × (trans2 + trans + 2)− j + Li : Li < j ≤ (Li + δ)

0 : j > (Li + δ)

(4.15)

where, Li is the current performance state of processor i in a multicore environment with a total

of n cores each capable of m performance states and the transition direction trans estimated to

be either −1, 0 or +1.

In order to help understanding of the consequence of Equation (4.15), an example is

considered describing the matrix for one of the three possible values of transition direction

trans = +1, 0,−1. A hypothetical multicore with a total of 4 cores (n = 4) each capable

of 5 performance states (m = 5) is considered. The current layout L is assumed to be L =

[0, 1, 2, 3], implying that processors C0, C1, C2 and C3 are in performance states P0, P1, P2

and P3 respectively. For all the three variations, the iteration delta constraint is assumed to be

δ = 2.

Evaluating the Manhattan matrix given in Equation (4.15) for δ = 2, n = 4, m = 5 and

32

L = [0, 1, 2, 3] and all values of trans,

Strans=+1 =

10 9 8 0 0

4 10 9 8 0

3 4 10 9 8

0 3 4 10 9

(4.16)

Strans=0 =

10 4 3 0 0

4 10 4 3 0

3 4 10 4 3

0 3 4 10 4

(4.17)

Strans=−1 =

10 4 3 0 0

9 10 4 3 0

8 9 10 4 3

0 8 9 10 4

(4.18)

is obtained. The following observations and reasoning can be made about these variations of

the Manhattan matrix:

• For each row, i, the highest weight of 2m (10), is assigned to the element Sij when

j = Li.

◦ As L0 = 0, the element S00 = 10.

◦ The highest weight is given to a transition favoring a processor to maintain its

current performance state if required.

• For each row, i, elements Sij are assigned a value of 0, when they are more that δ

elements away (|j − Li| > δ) from the current performance state.

◦ Elements S03 and S04 are assigned a value of 0.

◦ This honors the iteration delta constraint δ by assigning a weight (probability)

equal to zero to such transitions.

33

• For each row, i, and column Li < j ≤ (Li + δ), integer weights are assigned to Sij

such that they are in decreasing order starting from 2m − 1 (9) for trans = +1 or

starting from m− 1 (4) for trans = 0 and trans = −1.

◦ Weights for transitions linearly reduce as they are further from the current perfor-

mance state.

◦ Strans=+1: Elements S01, S02 are given values 9 and 8 respectively and weights

linearly reduce from 2m− 1 (9) as transition direction trans = +1 indicates that

mutations (DVFS reconfigurations) must favor transitions increasing the perfor-

mance of the multicore.

◦ Strans=0,−1: Elements S01, S02 are given values 4 and 3 respectively and reduce

from m − 1 for trans = −1, 0 as mutations must not favor transitions increas-

ing the power consumption of the multicore system for trans = −1 or avoid

transitions for trans = 0.

• For each row, i, and column (Li− δ) ≤ j < Li, integer values are assigned to Sij such

that they are in decreasing order starting from 2m − 1 (9) for trans = −1 or starting

from m− 1 (4) for trans = 0 and trans = +1.

◦ Weights for transitions linearly reduce as they are further from the current perfor-

mance state.

◦ Strans=−1: Elements S21, S20 are given values 9 and 8 respectively and weights

linearly reduce from 2m − 1 (9) as transition direction trans = −1 indicates

that mutations (DVFS reconfigurations) must favor transitions reducing the power

consumption of the multicore.

◦ Strans=0,+1: Elements S21, S20 are given values 4 and 3 respectively and reduce

from m − 1 for trans = +1, 0 as mutations must not favor transitions reducing

the performance of the multicore system for trans = +1 or avoid transitions for

34

trans = 0.

4.5.3 The Manhattan weight vector

The Manhattan weight vector W

∀j ∈ N : j < m;Wj =n−1∑i=0

Sij (4.19)

is constructed where n is the total number of cores and m is the total number of performance

states avaliable in each core. The Manhattan weight vector is achieved by summing along each

column of the Manhattan matrix and hence providing an insight into the locality of each state. A

higher value of Wi indicates a higher probability that there exists active processors in or around

performance state i, while a null value, Wi = 0 indicates that the performance state i can never

be achieved under the current delta constraint.

4.5.4 Cooperative demand distribution: The demand field

With lower values of ∆ (or with respect to the current iteration, δ), there is a possibility

that a state i, could be requested with a high core demand (demandi) but due to the current

layout and delta constraint, have a Manhattan weight Wi = 0, implying that a core in perfor-

mance state i can never be provided. Thus a method was developed in transforming the vector

demand in such a way that such null performance states cooperatively give up their demand to

the performance state closest to them which has a possibility of being selected. This procedure

is described in Algorithm 1.

The demand field F is essentially a replicate of demand, and differs under circumstances

when, for a particular performance state i, Wi = 0 and demandi > 0, then its demand is dis-

tributed to a friend f which varies based on the transition direction trans. If transition direction

trans = 1, then the demand is given to the state f closest and lesser than i with Wf > 0. If the

35Input: Demand vector demand, Manhattan weight vector WOutput: Demand field F, Proxy vector Proxyforeach Performance state i do

Proxyi = i ;Fi = demandi ;

endforeach Performance state i do

if Wi = 0 and demandi > 0 thenif trans = 1 then

f = max{x : x ∈ N and x < i and Wx > 0} ;endif t = −1 then

f = min{x : x ∈ N and x > i and Wx > 0} ;endif t = 0 then

f = x : x ∈ N and x ≈ i and Wx > 0 ;f is closest to i such that weight Wf > 0.

endFf = Ff + Fi ;Fi = 0 ;Proxyf = i ;

endend

Algorithm 1: Demand field computation

transition direction trans = −1 then a friend is searched in the opposite direction withWf > 0.

Finally when trans = 0, the closest state f to i with Wf > 0 is chosen as the friend. Once the

friend state f is computed, demandi is transferred to f. In order to track contributions from other

performance states, a vector Proxy is maintained indicating the source of the demand. Under

normal circumstances, Proxyi = i.

It must be noted that if the LEA system (Chapter 3) is used, this algorithm will cause

F = demand as the demand will always be within range of 1 from the current performance

state of each processor. The algorithm comes into play with the SEA system.

4.5.5 Greedy performance state selection

Once W and F are computed, performance state w is selected as the winning state when

the value of the product Ww × Fw is maximum and hence reacting to demand and maintaining

36

locality. the winning state w

w = arg maxx

Wx × Fx (4.20)

4.5.6 Greedy processor selection and transition

With the winning performance state selected, the processor to be transitioned is deter-

mined by the row c

c = arg maxx

Sxw ×Xx (4.21)

in the Manhattan matrix whose value Scw × Xc is maximum. Once the selection is made,

processor c is transitioned to performance state w by invoking the seeker cpufreq governor

described in Section 4.2. An addition, not mentioned in Figure 4.4 in order to maintain clarity,

is to abort the iteration algorithm if the value Scw = 0. This condition implies that there are no

more available processors which are capable of transitions due to the iteration delta constraint.

4.5.7 Parameter change and termination conditions

At the end of every iteration, the parameters δ, Lc and demandw are updated to represent

the transition. First as a transition was previously performed, the iteration delta constraint must

reduce

δ = δ − |Lc − w| (4.22)

by an equal order as a part of the iteration delta constraint δ is consumed by the current transi-

tion. Second, as processor c is no longer in performance state Lc, it is updated to it’s new value

w

Lc = w (4.23)

thus future iterations are aware of the current locality. Third, as a processor is assigned to

37

performance state w, the corresponding demand

demandProxyw = demandProxyw − 1 (4.24)

can be decremented. Note that, due to the distortion introduced by the cooperative demand dis-

tribution, demandProxyw is updated instead of demandw. Lastly, the poison vector’s position

for processor c

Xc = 0 (4.25)

is set to zero and hence disabling processor c in participating in a future transitions. It can be

observed from Equation (4.21) that a value of 0 for Xi will ensure that i will never be used as

the resulting value of Xi × Siw will be zero and will never be selected, achieving the desired

behavior.

This concludes a single iteration of the greedy direction based algorithm and is repeated

for Load iterations as long as the iteration delta constraint δ > 0. The upper bound on the

number of iterations is Load (Which can be at most n) and ensures early termination of the

iterative algorithm when unnecessary.

The algorithmic complexity of the entire procedure assuming n >> m, can be evaluated

to be O(n2), as the number of iterations can be at most n, and each iteration involves the eval-

uation of the Manhattan matrix which can have up to n rows. Note that the complexity of the

described procedure is the same as that of the dynamic programming approach (with memo-

rization) but is more efficient due to the consideration of transition direction. The complexity is

affordable as this is executed at a much coarser interval (mutation interval), if not, the mutation

interval can be increased to reduce the effects of the evaluation time.

Chapter 5

Experimental setup and results

A Linux kernel module as shown in Figure 5.1 was developed incorporating the ideas

discussed in chapters 3 and 4. The scheduling routines in the Linux kernel (To be specific,

schedule and switch to) were extended using kprobes [20] to add the performance directed

scheduling features. The choice to use kprobes [20] a feature essentially used for debugging

and instrumentation from the context of a kernel module pose two possible problems. One,

kprobes by themselves introduce unnecessary code paths and interrupts causing some if not

significant overhead (Less than 3%). Secondly, certain functions relating to migration are not

exported to kernel modules and hence indirect and suboptimal procedures were chosen to get the

necessary functionality. In spite of these disadvantages, having the project as a kernel module

accelerated the development process.

For a system with m total performance states and n total processors, ∆ can at most take

a value of n× (m− 1). Experiments were carried on a quad-core AMD Opteron (Barcelona),

and a patched version of the Linux 2.6.28 kernel. The logging interface as shown in Figure 5.1,

collects task specific and system wide statistics which enable in studying the behavior of both

the performance directed scheduler and the delta constrained mutator. The statistics collected by

the logging system aided in the computation of total energy consumption of each processor and

workload. In order to measure percentage slowdown, the six workloads mentioned in Chapter 2,

were run with full clock speed, recording their cumulative execution time. The power values

39

mentioned in [21] (Provided in Appendix A for convenience) were utilized to compute the

average power and energy consumption. Appendix B displays the adaptation time line graph

showcasing each of the two schedulers along with the delta constrained mutator.

The system call interface allows an application runtime (or an application) to provide

hints to the scheduler on the performance state demanded and allows compiler and runtime

power optimizing frameworks to interact with the underlying kernel based power optimizer to

create a more robust system. The system call interface is beyond the scope of this thesis and will

not be further discussed, but shown only as an example of possible extensions to the discussed

power management system.

Figure 5.1: The Seeker infrastructure

40

5.1 Trends along delta and interval

Two parameters namely the delta constraint (∆) and the mutation interval were intro-

duced in Chapter 4. In order to further study the system it is important to define the variation of

three important effects, namely, slowdown and power savings and energy efficiency improve-

ment as a function of these parameters. A fully factorial experiment was conducted varying

delta from 1 through 16 and the interval from 125ms through 1000ms. All the six workloads

in Table 2.2 (Chapter 2) were run for each experiment. Slowdown for each workload was com-

puted by comparing it’s execution time to that when run on a multicore system with all cores

executing with the maximum frequency. Power savings was computed by comparing the aver-

age power consumption to the highest being 115.0W per core. And finally Energy efficiency

improvement was computed by comparing the EPI of each experiment to that when all cores

are executing with the maximum possible clock speed.

Figure 5.2 shows the trends observed with respect to slowdown for the LEA and SEA

scheduling systems. An unnaturally high slowdown can be observed for small values of delta

(∆) which can be attributed to the plasticity in adaptation. The interval of mutation has little

significance towards slowdown for delta values higher than 3.The mutation interval begins to

play a vital role in differentiating the experiments based on slowdown once the delta constraint

is lowered beyond a value of 3. It can be observed that at lower values of delta, reducing

the mutation interval has an effect of decreasing the slowdown and effectively reducing the

performance lost. Comparing Figure 5.2(a) with 5.2(b) shows that lower values of delta have a

more profound negative effect on the SEA system than the LEA system. At ∆ = 1, the SEA

system exhibits at least 5% higher slowdown but both rapidly falls to the plateau close to 10%

beyond a delta constraint of 4.

The logging interface was configured to sample the executing workload’s characteris-

tic and namely the execution time and current performance state (which was used to arrive

41

(a) LEA

(b) SEA

Figure 5.2: Variation of average slowdown with delta and interval for the ladder (a) and theselect (b) estimation approach

at the power consumption by utilizing Table A.1) further utilized to compute the total en-

ergy consumption and finally the average power consumption (Average power consumption =

Total energy consumptiontotal execution time ). Percentage power savings was estimated by comparing this value with the

maximum possible power consumption of 115.0W (At the maximum possible clock speed of

2200MHz taken from Table A.1).

Figure 5.3 shows the variation of percentage power savings with varied values of delta

(∆) and mutation interval. A trend similar to percentage slowdown is observed for power sav-

42

ings: A high value of power savings for lower values of delta which plateaus at the value of

∆ = 3. Even though high power savings are desirable, the effective slowdown accompanying

lower delta constraints can be disadvantageous. A trend unique to power savings is the con-

tinued effect of mutation interval, whose decrease deteriorate power savings marginally for all

values of delta. During the experimentation, mutation intervals lower than 100ms was observed

to cause system instabilities and frequent crashes and hence the study was performed only for

values starting from 125ms. Even though this phenomenon can be attributed to the implemen-

tation, [13] warns system designers to avoid rapid DVFS transitions and should not be ignored.

The SEA system can be observed to be largely affected by delta. A sharp decline in power

savings can be observed for the SEA system which also provides lower power savings.

Figure 5.4 shows the variation of percentage improvement in energy efficiency (improve-

ment in observed EPI) from the same experiment with all processors executing with the maxi-

mum clock speed. Figures 5.2 and 5.3 indicate the weakness of the SEA system in comparison

to the LEA system. Going by the maximum values of energy efficiency improvement, it can be

noted that the SEA system does better by a margin of 5%. The plot for the SEA system sports an

even plain indicating the independence to the parameters: delta and interval. The LEA system

shows a maximum at the point corresponding to a delta value of 8 and an interval of 500ms but

the SEA system does better with lower values of interval and has an minimum line at an interval

value of 250ms. The LEA system deteriorates at a delta value of 16 for lower interval values

(probably relating to unnecessary elasticity exhibited by the system).

Based on observations made by the trends of power savings, slowdown and energy effi-

ciency improvement, it can be concluded that a value of delta ∆ = 4 provides the best among all

the compromises and the interval can be changed based on the arrival rate and average execution

times of typical jobs on the server or compute node. Having a very low mutation interval can

hamper potential power savings for systems with a high mean time between execution work-

loads hence advising system designers to consider all parameters before deploying the delta

constrained mutator. Average job duration can also affect possible deployments with respect

43

(a) LEA

(b) SEA

Figure 5.3: Variation of average power savings with delta and interval for the ladder (a) and theselect (b) estimation approach

to mutation interval. Care must be taken to choose the mutation interval as a fraction of the

average execution times of typical jobs running on the system.

5.2 Trends along maximum allowed mutation rate

Figures 5.2, 5.3 and 5.4 shows the variation of slowdown, average power savings and

energy efficiency improvement respectively as a function of mutation interval and the delta

44

(a) LEA

(b) SEA

Figure 5.4: Variation of average energy efficiency (EPI) improvement with delta and intervalfor the ladder (a) and the select (b) estimation approach

constraint. A parameter maximum allowed mutation rate is the amalgamation of these two

parameters describing the rate at which mutations are allowed in the system expressed as muta-

tions per second.

Maximum allowed mutation rate =∆

Mutation Interval(5.1)

45

Even though less descriptive for a system administrator in choosing the parameters for deploy-

ment, this view of the results presented in Figures 5.2, 5.3 and 5.4 can be expressed as a function

of the maximum mutation rate and shown in Figure 5.5 and making the trends easier to observe.

Both the LEA and SEA systems show that increasing the maximum allowed mutation rate above

20 causes the slowdown to become less prominent; a direct effect of more elasticity of the adap-

tation system. It can also be observed that further increase in the maximum allowed mutation

rate has no significant effect of reducing the slowdown or improving the power savings. The

energy efficiency sports a bell curve for the LEA system reaching a maximum of approximately

2.5% at 40. As observed with Figure 5.4, a flat energy efficiency improvement variation is ob-

served for the SEA system consistent at 2.5% and slightly dipping around maximum allowed

mutation rate equal to around 45.

(a) LEA (b) SEA

Figure 5.5: Variation of average power savings, slowdown and energy efficiency improvementwith maximum mutation rate for the LEA (a) and the SEA (b) scheduling systems

5.3 Comparing various methodologies and varying workloads

Chapter 3 described two methods of evaluating the performance state and hence pro-

viding two distinct methods of scheduling which affects the adaptation of the delta system.

46

Chapter 4 introduced a common power management system, the ondemand power optimizer.

Experiments were conducted varying the mutation interval from 125ms to 1000ms for all three

systems. Section 5.1 recommends the minimum value of delta ∆ = 4 in order to minimize

the effects of slowdown due to adaptation plasticity and hence the delta mutator with a value of

∆ = 4 was utilized for the experiments proposed in this section. Utilizing the logging interface,

the execution sample performance state (and in turn the power consumption), execution time

and retired instructions were recorded to later compute the average power consumption for each

workload, the energy efficiency, measured as energy consumption per executed instruction and

slowdown. These experiments were repeated for each of the power management systems:

• Ondemand governor, mutation interval: 125ms, 250ms, 500ms, 1000ms

• The ladder performance directed scheduling system with the delta constrained mutator

with a delta value, ∆ = 4 and mutation interval: 125ms, 250ms, 500ms, 1000ms

• The select performance directed scheduling system with the delta constrained mutator

with a delta value, ∆ = 4 and mutation interval: 125ms, 250ms, 500ms, 1000ms

For each experiment, the percentage power savings, percentage slowdown and EPI (energy per

instruction) improvement was computed and tabulated. These experiments were clustered based

on workload in order to further categorize the results.

Due to the adaptive nature of the power management mechanism and the high depen-

dence on the IPC characteristic of workloads, it can be hypothesized that workloads with lower

IPC (The Low workload) should expect the maximum power savings while, workloads with

large IPC (The High workload) should expect the minimum slowdown and power savings.

Figure 5.6 shows the average values of the parameter space for each workload under

varying mutation intervals for the LEA system and the delta mutation engine. The High work-

load suffers from an insignificant value of slowdown (around 1%) but also gains no power

savings. There is a small deterioration of energy efficiency which can be perceived to be in-

significant. Meanwhile, the Low workload achieves an excess of 40% power savings with a

47

meager 20% of performance loss and a considerable gain in energy efficiency of 25%. These

two workloads show an agreement with the hypothesis made above. The PLow-Low shows a

drastic amount of slowdown (30%) and the most affected in the group. The median slowdown

for the LEA system can be estimated to be around the 16% margin with a median power sav-

ings of 12%. Three out of 6 (Low, Low-High and PLow-Low) show improvement in energy

efficiency.

Figure 5.6: Variation of power savings, energy efficiency improvement and slowdown with eachworkload for the ladder estimation approach (LEA) with the delta mutation engine at ∆ = 4

Figure 5.7 shows the average values of the parameter space for each workload under

varying mutation intervals for the SEA system and the delta mutation engine. Similar observa-

tions can be made to the SEA system as LEA, except that the SEA system does worse in terms

of slowdown ranging from a minimum of 4% (High workload) to a very high 35% (PLow-Low

workload). The power savings achieved by the SEA system is considerably lower than the LEA

system ranging from a 1% to 28%. Even though worse in terms of slowdown and power sav-

ings, the SEA system does better in terms of energy efficiency improvement with 4 out of 6

48

(Low, Low-High, PHigh-PLow and PLow-Low) workloads achieving a positive value and the 2

other workloads (High and PLow-High) show meager deterioration. The shocking increase in

slowdown can possibly be attributed to a wrong choice of thresholds mentioned in Chapter 3,

further investigations must be made with varying threshold values.

Figure 5.7: Variation of power savings, energy efficiency improvement and slowdown with eachworkload for the select estimation approach (SEA) with the delta mutation engine at ∆ = 4

Finally, Figure 5.8 shows the average values of the parameter space for each workload

under varying mutation intervals for the Ondemand system. There is an insignificant (less than

0.5%) amount of achievable power savings while still susceptible to varied amounts of slow-

down ranging from nothing to 5%. Even though by theory the ondemand system should never

experience slowdown, the observed values can best be attributed to the overhead suffered from

the particular implementation of the scheduling system explained in the introductory paragraph

implying that a production implementation of the scheduling and delta mutation system can re-

duce the suffered slowdowns described above by a margin of 5%. Insignificant improvements in

energy efficiency is observed. The PLow-Low workload suffers the minimum amount of slow-

49

down indicating characteristics dissimilar to that of the other workloads, this is also observed

from the graphs for the LEA and SEA systems.

Figure 5.8: Variation of power savings, energy efficiency improvement and slowdown with eachworkload for the Ondemand governor

Any power management system will impose varied amounts of slowdown on the exe-

cuting workloads based on the design. The experiments showcase the return on the slowdown

achieved by the proposed system with significant power savings ranging from 1% to 40%. But

the observed slowdown indicate room for improvement in the scheduling and mutation systems.

Figures 5.6, 5.7 and 5.8 show that any amount of power savings are also accompanied by var-

ious degrees of slowdown. The goal is to maximize power savings while minimizing suffered

slowdown. Appendix B shows the mutation time-line of all the workloads with both LEA and

SEA systems which should provide insight into the time-line mutation process.

Chapter 6

Future work

The scheduling methodology can integrate the work presented in [22] for higher accu-

racy in predicting the performance state required by a particular task. Further analysis on the

relationship of clock speed invariance of workloads may prove valuable in improving the select

scheduling system and deriving a more optimal methodology of selecting its threshold values.

The fact that there exists a correlation between IPC and speedup achievable with increased

clock speed, can be further utilized to produce a more robust system. The first steps would be

to provide strong support within the Linux kernel for asymmetric multicore processors which

will enable further study into the aspect of performance directed scheduling.

A system call interface was integrated into the existing system which can be utilized

to develop compiler or runtime frameworks, which would potentially have more information

about program characteristics in developing a hybrid system to provide higher power savings

with lower performance loss.

Chapter 7

Summary and Conclusion

This thesis presented an approach of scheduling tasks on multicore processors with re-

spect to performance requirements and performing a constrained system wide dynamic voltage

and frequency transitions in reaction to the demands of the performance characteristic of the

executing workloads. A novel methodology was presented in controlling the magnitude of

these transitions to improve stability and longevity of the multicore system. The methodology

was shown to possess characteristics to minimize slowdown of workloads while improving the

power savings achieved.

Workloads were shown to possess characteristics to possibly make it immune to clock

speed improvements thus enhancing the motivation of a performance determined power man-

agement system to conserve power during system active state. Experiments with various static

layouts were conducted to characterize and showcase the performance directed scheduler. It

was demonstrated that a statically assigned performance state layout of a system can prove

to drastically deteriorate performance of the system in an environment with non-deterministic

workload sets. Two scheduling systems, the ladder and select, were developed to schedule tasks

based on their performance characteristic.

A system was developed to allow a performance directed scheduler to direct an asyn-

chronously run power optimizer to manage the performance states of each core in an otherwise

homogeneous multicore environment. The problem was shown to be NP-Hard and an efficient

52

algorithm was developed in solving the Multiple-choice knapsack problem as a direction based

iterative greedy algorithm.

Experiments were conducted to measure slowdown and power savings along the sys-

tem’s entire parameter space to determine the most favorable constraint on the magnitude of the

allowed DVFS transitions. This was utilized to compare the power management system with a

popular load based power optimizer, the ondemand governor. The combination of the perfor-

mance directed scheduler and the delta constrained mutator was shown to achieve higher power

efficiency and better power savings with marginal slowdown.

Bibliography

[1] W. Kim, M. S. Gupta, G.-Y. Wei, and D. Brooks, “System level analysis of fast, per-coredvfs using on-chip switching regulators,” in 2008 IEEE 14th International Symposiumon High Performance Computer Architecture. Piscaatway, NJ, USA: IEEE, 2008, pp.123–134.

[2] K. M. P. R. M. K. M. J. Irwin, “Phase-aware adaptive hardware selection for power-efficient scientific computations,” in International Symposium on Computer Architecture/Proceedings of the 2007 international symposium on Low power electronics and design.Portland, OR, USA: ACM, 2007, pp. 403–406.

[3] M. Kondo, H. Sasaki, and H. Nakamura, “Improving fairness, throughput and energy-efficiency on a chip multiprocessor through dvfs,” vol. 35, no. 1. New York, NY, USA:ACM, 2007, pp. 31–38.

[4] Q. Wu, M. Martonosi, D. W. Clark, V. J. Reddi, D. Connors, Y. Wu, J. Lee, andD. Brooks, “A dynamic compilation framework for controlling microprocessor energy andperformance,” in MICRO 38: Proceedings of the 38th annual IEEE/ACM InternationalSymposium on Microarchitecture. Washington, DC, USA: IEEE Computer Society,2005, pp. 271–282.

[5] H. Sasaki, Y. Ikeda, M. Kondo, and H. Nakamura, “An intra-task dvfs technique basedon statistical analysis of hardware events,” in Conference On Computing Frontiers/Proceedings of the 4th international conference on Computing frontiers. Ischia, Italy:ACM, 2007, pp. 123–130.

[6] A. Rangasamy, R. Nagpal, and Y. Srikant, “Compiler-directed frequency and voltage scal-ing for a multiple clock domain microarchitecture,” in CF ’08: Proceedings of the 5thconference on Computing frontiers. New York, NY, USA: ACM, 2008, pp. 209–218.

[7] B. Rountree, D. K. Lownenthal, B. R. de Supinski, M. Schulz, V. W. Freeh, and T. Bletsch,“Adagio: making dvs practical for complex hpc applications,” in ICS ’09: Proceedings ofthe 23rd international conference on Supercomputing. New York, NY, USA: ACM, 2009,pp. 460–469.

[8] B. Rountree, D. K. Lowenthal, S. Funk, V. W. Freeh, B. R. de Supinski, and M. Schulz,“Bounding energy consumption in large-scale mpi programs,” in SC ’07: Proceedings ofthe 2007 ACM/IEEE conference on Supercomputing. New York, NY, USA: ACM, 2007,pp. 1–9.

54

[9] K. Meng, R. Joseph, R. P. Dick, and L. Shang, “Multi-optimization power managementfor chip multiprocessors,” in Proceedings of the 17th international conference on Parallelarchitectures and compilation techniques. Toronto, Ontario, Canada: ACM, 2008, pp.177–186.

[10] C. Isci, G. Contreras, and M. Martonosi, “Live, runtime phase monitoring and predictionon real systems with application to dynamic power management,” in In: Proceedings ofthe 39th International Symposium on Microarchitecture (MICRO-39). Los Alamitos,CA, USA: IEEE Computer Soc, 2006, p. 12pp.

[11] Y.-H. Lu, L. Benini, and G. D. Micheli, “Operating-system directed power reduction,” inConference On Computing Frontiers/ Proceedings of the 4th international conference onComputing frontiers. Rapallo, Italy: ACM, 2000, pp. 37–42.

[12] C. Boneti, R. Gioiosa, F. J. Cazorla, and M. Valero, “A dynamic scheduler for balanc-ing hpc applications,” in SC ’08: Proceedings of the 2008 ACM/IEEE conference onSupercomputing. Piscataway, NJ, USA: IEEE Press, 2008, pp. 1–12.

[13] N. Soundararajan, N. Vijaykrishnan, and A. Sivasubramaniam, “Impact of dynamicvoltage and frequency scaling on the architectural vulnerability of gals architectures,”in ISLPED ’08: Proceeding of the thirteenth international symposium on Low powerelectronics and design. New York, NY, USA: ACM, 2008, pp. 351–356.

[14] Standard Performance Evaluation Corporation, “The SPEC CPU 2006 benchmark suite,”2006. [Online]. Available: http://www.spec.org

[15] T. Li, D. Baumberger, D. A. Koufaty, and S. Hahn, “Efficient operating system schedulingfor performance-asymmetric multi-core architectures,” in SC ’07: Proceedings of the 2007ACM/IEEE conference on Supercomputing. New York, NY, USA: ACM, 2007, pp. 1–11.

[16] D. P. Gulati, C. Kim, S. Sethumadhavan, S. W. Keckler, and D. Burger, “Multitaskingworkload scheduling on flexible-core chip multiprocessors,” in Proceedings of the 17thinternational conference on Parallel architectures and compilation techniques. Toronto,Ontario, Canada: ACM, 2008, pp. 187–196.

[17] R. Nathuji and K. Schwan, “Virtualpower: coordinated power management in virtualizedenterprise systems,” in SOSP ’07: Proceedings of twenty-first ACM SIGOPS symposiumon Operating systems principles. New York, NY, USA: ACM, 2007, pp. 265–278.

[18] V. Pallipadi and A. Starikovskiy, “The on-demand governor,” in 2006 LinuxSymposium, 2006. [Online]. Available: http://ftp.funet.fi/pub/Linux/kernel/people/lenb/acpi/doc/OLS2006-ondemand-paper.pdf

[19] J. C. Bean, “Multiple choice knapsack functions,” in Proceedings of the 2007 InternationalSymposium on Low-Power Electronics and Design (ISLPED’07), Department of Indus-trial operations and Engineering, University of michigan, Ann Arbor, Michigan 48109,1987.

[20] R. Krishnakumar, “Kernel korner: kprobes-a kernel debugger,” Linux J., vol. 2005, no.133, p. 11, 2005.

55

[21] AMD, AMD Family 10h Server and Workstation Processor Power and Thermal DataSheet, June 2009. [Online]. Available: http://support.amd.com/us/Processor TechDocs/43374.pdf

[22] M. Curtis-Maury, A. Shah, F. Blagojevic, D. S. Nikolopoulos, B. R. de Supinski, andM. Schulz, “Prediction models for multi-dimensional power-performance optimization onmany cores,” in Proceedings of the 17th international conference on Parallel architecturesand compilation techniques. Toronto, Ontario, Canada: ACM, 2008, pp. 250–259.

Appendix A

AMD Opteron capabilities

Performance State Clock Speed (MHz) Power Con-sumption perProcessor Core(Watts)

0 1100 64.61 1400 75.62 1700 88.83 2000 108.24 2200 115.0

Table A.1: Correlation between performance states with clock speed and power consumption

Appendix B

Mutation time-line per workload

For each figure, the X-axis is time with a granularity of a mutation interval. The Y-axis is

the stacked processor count for each performance state. Thus for each vertical slice, the width of

each color, corresponds to the total number of cores executing with that particular performance

state.

(a) Ladder (b) Select

Figure B.1: Adaptation time-line for the High workload with the ladder (a) and select (a)scheduling

58

(a) Ladder (b) Select

Figure B.2: Adaptation time-line for the Low workload with the ladder (a) and select (a)scheduling

(a) Ladder (b) Select

Figure B.3: Adaptation time-line for the Low-High workload with the ladder (a) and select (a)scheduling

59

(a) Ladder (b) Select

Figure B.4: Adaptation time-line for the PHigh-PLow workload with the ladder (a) and select(a) scheduling

(a) Ladder (b) Select

Figure B.5: Adaptation time-line for the PLow-Low workload with the ladder (a) and select (a)scheduling

60

(a) Ladder (b) Select

Figure B.6: Adaptation time-line for the PLow-High workload with the ladder (a) and select (a)scheduling

Appendix C

Source code

All the source code has been uploaded in a public git repository and available at:

git://github.com/amithash/seeker-scheduler.git.

Note that in order to checkout the source code, you will need to have the git source

control management utilities installed.