self-adaptive workload classification and forecasting for proactive resource provisioning · 2013....

KIT – University of the State of Baden-Wuerttemberg and

National Research Center of the Helmholtz Association

SOFTWARE DESIGN AND QUALITY GROUP

INSTITUTE FOR PROGRAM STRUCTURES AND DATA ORGANIZATION, FACULTY OF INFORMATICS

www.kit.edu

Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning

Nikolas R. Herbst, Nikolaus Huber,

Samuel Kounev, Erich Amrehn (IBM R&D) ICPE 2013, Prague

Motivation

history now near future

wo

rklo

ad

in

ten

sity

Motivation Foundations Approach Evaluation Conclusion

Nikolas R. Herbst Workload Classification and Forecasting ICPE 2013, Prague

Motivation


wo

rklo

ad

in

ten

sity

Forecast approaches of

the time series analysis:



Motivation


wo

rklo

ad

in

ten

sity

Forecast approaches of

the time series analysis:

Idea: Intelligent and dynamic use of

different tools out of the toolkit

Goal: Providing information on most

likely future developments

„Knowing about a problem before running into it”



Workload Classification & Forecasting (WCF)

Self-adaptive Mechanism

Overhead Limitation

Forecast Frequency

Forecast Horizon

Confidence Level

Forecast Objectives





Overhead Limitation

Forecast Frequency

Forecast Horizon

Confidence Level

Forecast Objectives

INITIAL

Naïve,

Smoothing,

Random

Walk

Forecast Strategy Overhead Groups

FAST

Trend

Inter-

polation

COMPLEX Decom-

position &

Seasonal

Patterns





Overhead Limitation

Forecast Frequency

Forecast Horizon

Confidence Level

Forecast Objectives

(I) Select Strategies

Forecast Objectives

Workload Intensity

Trace

INITIAL

Naïve,

Smoothing,

Random

Walk


FAST

Trend

Inter-

polation

COMPLEX Decom-

position &

Seasonal

Patterns





Overhead Limitation

Forecast Frequency

Forecast Horizon

Confidence Level

Forecast Objectives

(I) Select Strategies

Forecast Objectives

Workload Intensity

Trace

INITIAL

Naïve,

Smoothing,

Random

Walk


(II) Forecast

& Evaluate

Accuracy

Accuracy

Feedback

FAST

Trend

Inter-

polation

COMPLEX Decom-

position &

Seasonal

Patterns



Time Series Analysis

[BFAST]



Forecasting Strategies Survey

Basic Methods (initial)

Naïve, Moving Averages, Random Walk






Trend Interpolation (fast)

Simple Exponential Smoothing (SES) [Hynd08]

Cubic Smoothing Splines [Hynd02]

Croston‘s method for intermittent time series [Shen05]

Autoregressive Moving Averages (ARMA11) [Box08]






Estimation and Modelling of Seasonal Pattern (complex)

Extended Exponential Smoothing (ETS) [Hynd08, Hyn08]

ARIMA framework with automatic model selection [Box08, Hynd08]

tBATS for complex seasonal patterns [Live11]

Trend Interpolation (fast)

Simple Exponential Smoothing (SES) [Hynd08]

Cubic Smoothing Splines [Hynd02]

Croston‘s method for intermittent time series [Shen05]

Autoregressive Moving Averages (ARMA11) [Box08]



Forecast Accuracy Metric

Mean absolute scaled error (MASE) [Hynd06]

Observation

Forecast

0 1 2 … n time[t]

va

lue

s e3


𝑒𝑡 = 𝑓𝑜𝑟𝑒𝑐𝑎𝑠𝑡𝑉𝑎𝑙𝑢𝑒𝑡 − 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑𝑉𝑎𝑙𝑢𝑒𝑡

𝑏𝑛 =1𝑛 × |𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑𝑉𝑎𝑙𝑢𝑒𝑖 − 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑𝑉𝑎𝑙𝑢𝑒𝑖−1|

𝑛

𝑖=1

𝑚𝑎𝑠e 0; 𝑛 = 𝑚𝑒𝑎𝑛𝑡={1;𝑛}(|𝑒𝑡𝑏𝑛|)


Forecast Accuracy Metric

Mean absolute scaled error (MASE) [Hynd06]

Observation

Naive Forecast

Observation

Forecast

0 1 2 … n time[t]

◄ Forecast

Reference ►

va

lue

s

va

lue

s

0 1 2 … n time[t]

e3


𝑒𝑡 = 𝑓𝑜𝑟𝑒𝑐𝑎𝑠𝑡𝑉𝑎𝑙𝑢𝑒𝑡 − 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑𝑉𝑎𝑙𝑢𝑒𝑡

𝑏𝑛 =1𝑛 × |𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑𝑉𝑎𝑙𝑢𝑒𝑖 − 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑𝑉𝑎𝑙𝑢𝑒𝑖−1|

𝑛

𝑖=1

𝑚𝑎𝑠e 0; 𝑛 = 𝑚𝑒𝑎𝑛𝑡={1;𝑛}(|𝑒𝑡𝑏𝑛|)


Workload Intensity Characterization

Workload Intensity Trace:

Time Series of Request Arrival Rates

Length

Gradient Mono-tonicity

Positivity

Variance

Frequency

Quartile Dispersion

Burstiness

High level data analysis to gain information on:

• Noise level & occurrences of unpredictable bursts

• Influence of trends and seasonal patterns


Data

Attributes

Metrics


Decision Tree for Classification



Experiment: Example for

Forecast Accuracy Improvement

• Real-world workload intensity trace: IBM CICS transactions on System z mainframe

• Comparison of WCF approach to ETS and Naïve forecast



Experiment

WCF

ETS

Naïve

WC

F |

ETS

| N

aive

percentage error



Case Study:

Example for Using Forecast Results

• Scenario: Additional server instances at certain thresholds, 3 weeks

• Real-world workload intensity trace (Wikipedia DE page requests per hour)



Case Study

Resource provisioning:

(I) Without forecasting (solely reactive):

Resource provisioning actions triggered by

76 SLA violations

(II) Interpreting WCF forecast results (add. proactive):

Reduction to 34 or less SLA violations

No significant change in resource usage observed

(server instances per hour)



Related Work

Forecasting for Proactive Resource

Provisioning

[Grun12, Hedw10, Gmac08, Mena04,

Bens07]

Time Series Analysis

[Box08, Hynd08 Shum11]

Self-Adaptive Resource

Provisioning

e.g. [Hube11, Koun10]

Workload & Performance Monitoring

e.g. [vHoo12]

Focus on forecasting of performance metrics not workload intensity

Focus on single tools of the toolkit or other toolkits no dynamic selection



Summary & Outlook

Survey on Forecast Approaches

Construction of a

Workload Classification Scheme Implementation of the

WCF-System

provides continuous forecast

results at run-time Experiments and Case Study:

Evaluation based on real-world

workload intensity traces

Forecast Accuracy

Improvement: > 37 % compared to ETS as an

established approach

Proactive Resource

Provisioning enabled: > Up to 75 % less SLA violations

than reactive

Future Work: > System Integration with Monitoring

> Filters: Objective Selection, Splitting

> Use for Anomaly Detection [Biel12]



Literature

[Bens07] M. Bensch, D. Brugger, W. Rosenstiel, M. Bogdan, W. G. Spruth, and P. Baeuerle, Self-

learning prediction system for optimization of workload management in a mainframe operating

system," in ICEIS 2007

[BFAST] R Package: BFAST, Breaks for additive Season and Trend, http://bfast.r-forge.r-project.org/

[Biel12] T. C. Bielefeld, Online Performance Anomaly Detection for Large-Scale Software Systems,

March 2012, Diploma Thesis, University of Kiel

[Box08] G. E. P. Box, G. M. Jenkins, and G. C. Reinsel, Time series analysis : forecasting and control,

2008

[Gmac07] D. Gmach, J. Rolia, L. Cherkasova, and A. Kemper, Workload analysis and demand prediction

of enterprise data center applications, IISWC '07

[Grun12] A. Amin, A. Colman, and L. Grunske, An Approach to Forecasting QoS Attributes of Web

Services Based on ARIMA and GARCH Models, ICWS 2012

[Hedw10] M. Hedwig, S. Malkowski, C. Bodenstein, and D. Neumann, Towards autonomic cost-aware

allocation of cloud resources, ICIS 2010

[Hube11] N. Huber, F. Brosig, and S. Kounev, Model-based Self-Adaptive Resource Allocation in

Virtualized Environments," SEAMS 2011

[Hynd02] R. J. Hyndman, M. L. King, I. Pitrun, and B. Billah, Local linear forecasts using cubic

smoothing splines, Monash University, Department of Econometrics and Business Statistics,

2002


Literature

[Hynd06] R. J. Hyndman and A. B. Koehler, Another look at measures of forecast accuracy,

International Journal of Forecasting

[Hynd08] R. J. Hyndman and Y. Khandakar, Automatic time series forecasting: The forecast package for

R 2008

[Hyn08] R. J. Hyndman, Koehler, Forecasting with Exponential Smoothing : The State Space

Approach, Springer Series in Statistics, Berlin, 2008

[Koun10] S. Kounev, F. Brosig, N. Huber, and R. Reussner, Towards self-aware performance and

resource management in modern service-oriented systems, SCC '10

[Live11] A. M. De Livera, R. J. Hyndman, and R. D. Snyder, Forecasting time series with complex

seasonal patterns using exponential smoothing, Journal of the American Statistical

Association, vol. 106, no. 496, pp.1513, 2011

[vHoo12] A. van Hoorn, J. Waller, and W. Hasselbring, Kieker: A framework for application performance

monitoring and dynamic software analysis," ICPE 2012

[Mena04] M. N. Bennani and D. A. Menasce, Assessing the robustness of self-managing computer

systems under highly variable workloads, 2004

[Shen05] L. Shenstone and R. J. Hyndman, Stochastic models underlying Croston's method for

intermittent demand forecasting," Journal of Forecasting, vol. 24, no. 6, pp. 389, 2005

[Shum11] R. H. Shumway, Time Series Analysis and Its Applications: With R Examples, Springer


Backup: Experiment WCF4 vs. tBATS, ARIMA

WCF limited to choose from tBATS and ARIMA

Significant accuracy improvement by combination and dynamic strategy selection


Backup:

Process of Classification & Forecasting

classification:

(I) select strategies

forecast

execution &

result output

forecast horizon/

frequency

(need not to be equal)

classification:

(II) evaluate selection

…

forecast

execution &

result output

new forecast

horizon

new classification:

(I) select strategies

forecast

execution &

result output

classification frequency (# forecast executions)

forecast

execution &

result output

forecast

execution &

result output

forecast

execution &

result output


Backup: Architecture and Implementation


Backup: Sequence Diagram


Backup: Class Diagram - Management

Nikolas R. Herbst Workload Classification and Forecasting ICPE 2013,

Prague

Backup: Class Diagram - Classification


Backup: Class Diagram – Forecasting


Backup: Forecast Objectives


Backup: Forecast Strategy Overhead Groups


self-adaptive workload classification and forecasting for proactive resource provisioning · 2013....

Documents