optimisation de l'ordonnancement de workflows …cerin/vichy2014/yassa...ow scheduling in cloud...

33
Context & Motivations Problem Description Proposed Solutions Experiments Conclusion & Perspectives EISTI Optimisation de l’ordonnancement de workflows bas´ e sur les QoS dans les environnements de Cloud Computing. Sonia Yassa Rachid Chelouah, Hubert Kadima, Bertrand Granado [email protected] ETIS/ENSEA Universit´ e de Cergy-Pontoise France L@RIS ´ Ecole Internationale des Sciences du Traitement de l’Information EISTI France Vichy 2014 - 04 juin 2014 Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 1 / 33

Upload: others

Post on 03-Aug-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Optimisation de l’ordonnancement de workflowsbase sur les QoS dans les environnements de Cloud

Computing.

Sonia Yassa

Rachid Chelouah, Hubert Kadima, Bertrand [email protected]

ETIS/ENSEA Universite de Cergy-Pontoise France

L@RIS Ecole Internationale des Sciences du Traitement de l’Information EISTIFrance

Vichy 2014 -04 juin 2014

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 1 / 33

Page 2: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Outline

Context & Motivations

Problem Description

Proposed Solutions

Experiments

Conclusion & Perspectives

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 2 / 33

Page 3: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Cloud Computing

• New paradigm.

• SaaS, PaaS, IaaS, HaaS, * as a services.

• Pay-as-you-go model!!

• QoS parameters : SLA (Service Level Agreement)

Figure: Cloud Computing Services Offering Models

• Cloud used to run users applications .

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 3 / 33

Page 4: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Applications

• Scientific workflow:bioinformatic, astronomy, ...

• Contain• Large number of tasks.• Complex data of various sizes.

• Need :• High computation power.• IT infrastructures availability.

• Require(QoS):• Execution time• Execution cost• Reliability

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 4 / 33

Page 5: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Workflow scheduling

• Workflow schedulingProcess that maps and managesthe execution of interdependenttasks on distributed resources.

• Related worksSeveral previous works:

• time and cost.• Reliability, Energy..

• ContributionExtending previous works tohandle multiple QoS.

Figure: Conflicting Objectives

• Workflow Scheduling =NP-Complete.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 5 / 33

Page 6: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Objectives

• Developing MOO algorithms for QoS-based workflowscheduling in cloud environments.

• The Scheduler:• Analyze the users QoS parameters,• Map the workflow tasks on resources so that:• The execution must be completed,• Users QoS constraints must be satisfied,• The use of cloud resources must be optimized.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 6 / 33

Page 7: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Cloud computing model

• M heterogeneous Virtual Machines (VMs),

• fully interconnected,

• Different processing capabilities,

• Different prices.

• DVFS-enabled in the case of HaaS.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 7 / 33

Page 8: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Workflow model

• Workflow = DAG G = (T, E)• T : tasks, and E : data dependencies

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 8 / 33

Page 9: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

QoS parameter models(1/3)

• Time model (makespan)

Ttotal = maxAFT (Texit)

• Cost model

Ctotal = Cex + Ctr

Cex =n∑

i=1

wi ∗ eci

Ctr =n∑

i=1

n∑j=1

dij ∗ trcij

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 9 / 33

Page 10: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

QoS parameter models (2/3)

• Reliability model

Probability that the workflow will beexecuted successfully.

Rtotal = exp −∑n

i=1 ωi∗λi

λi = failure rate of the VMi .

• Availability model

Average time that the VMs are idlesduring the workflow execution.

Atotal =1

M?(1−

M∑j=1

executionTimejMakespan

)

M= number of VMs.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 10 / 33

Page 11: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

QoS parameter models (3/3)

• Energy model

Derived from the power consumption model in CMOS circuits(DVFS )

Ec =n∑

i=1

ACef ∗ v2i ∗ fi ∗ wr∗i

• A: switches per clock cycle, C: capacitance load,

• V: supply voltage, f: frequency,

• wr: real completion time of task on the scheduled processor

Higher the frequency level = higher the energy consumption

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 11 / 33

Page 12: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Scheduling Model

• Construct a mapping M of tasks to VMs• Without violating precedence constraints• Minimizing the conflicting objectives:

1 Makespan: minTtotal(M)2 Cost: minCtotal(M)3 Reliability: maxRtotal(M)4 Availability: maxAtotal(M)5 Energy: minEtotal(M)

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 12 / 33

Page 13: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Proposed Solutions

• GA: Makespan + Cost + reliabilty + Availability.

• GAVI(Viral Infection): Makespan + Cost + reliabilty +Availability + QoS Constraints + hard constraints(Taski-Resourcej)

• NSGA-II

• PAES

• DVFS-MODPSO : Makespan + Cost + Energy.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 13 / 33

Page 14: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

The Proposed Solution

• The use of Genetic Algorithms (GA) is usually efficient toaddress NP-hard problem.

• Define:• A structure encoding a solution.• A fitness function.• Genetic operators: selection, crossover, mutation

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 14 / 33

Page 15: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Simplified GA flowchart

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 15 / 33

Page 16: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Solution encoding

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 16 / 33

Page 17: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Fitness function

• n QoS attributes to be minimized.

• m QoS attributes to be maximized.

F =n∑

i=1

ωi ? (qi − qimin

qimax − qimin) +

M∑j=1

ωj ? (1 −qj − qjmin

qjmax − qmin)

• ωi (ωj): weights for QoS attribute

• qimin, (qimax): minimal (maximal) values of the objective i inthe current pool of the solutions.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 17 / 33

Page 18: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Fitness function

Fitness = F ∗∏

Penaltyi

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 18 / 33

Page 19: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Genetic operators

• Selection• keep only the top a%

solutions.• Select parent solutions in the

remaining pool.

• Mutation• Each solution has Y% to be

mutated,• Change VM associated with

the task.

• Crossover

Ratio =Fitness1

Fitness1 + Fitness2

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 19 / 33

Page 20: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Genetic operators

• Viral Infection

• It aims at repairing solutions that do not respect the hardconstraints instead of deleting them.

• The idea is to create a virus for each constraint whose type is: Task X must be scheduled on Resource Y.

• After selection, crossover and mutation : every solution thatdoesnt meet the CSP requirements is infected by theassociated virus and repaired.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 20 / 33

Page 21: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Applications

• Benchmark employed in the simulations:

(a) Hybrid workflow(protein annotation)

(b) Parallel workflow(neuro-science)

Figure: Workflow applications

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 21 / 33

Page 22: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Cloud

• VM characteristics from Amazon EC2 cloud

• Evaluation process:

• Perform a single-objective optimization.• Run our algorithm to get the multi-objectives results of each

metric.• Compare the values obtained.• Vary the weights of the various QoS metrics.• Compare our results with those of the HEFT algorithm.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 22 / 33

Page 23: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Results

• Hybrid workflow best values • Parallel workflow best values

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 23 / 33

Page 24: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Results

• Hybrid workflow results • Parallel workflow results

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 24 / 33

Page 25: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Results

• Compare GAVI with GA by assessing the number of invalidsolutions.

• Experiment run on the two 15 tasks workflows using differentnumbers of constraints and a pool of 3 resources : Theefficiency of the classical genetic algorithm without viralinfection plummet as the number of constraints increases.

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 25 / 33

Page 26: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Results

• Interest of the proposed penalty method when addingconstraints on the QoS

• Results of the QoS constrained optimization on the parallelworkflow

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 26 / 33

Page 27: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Results

• comparing with other algorithms

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 27 / 33

Page 28: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Conclusions

• We have investigated the QoS based scheduling problem onscalable computing systems

• We proposed GA for workflow scheduling• A new fitness function model which can takes into account a

great number of users and providers QoS requirementsspecified in the SLA.

• The proposed solution outperforms related approaches .

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 28 / 33

Page 29: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Perspectives

• We plan to implement our algorithms in workflowmanagement systems (Cloudbus).

• Explore other meta-heuristics (e.g.,PSO+Tabu, ACO, SA)

• Consider other QoS metrics: Security,

• Parallelization

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 29 / 33

Page 30: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Publications(1/3)

• Journals:

1 S. Yassa, J. Sublime, R. Chelouah, H. Kadima, G-S. Jo, B.Granado, A Genetic Algorithm for Multi-ObjectiveOptimization in Workflow Scheduling with Hard Constraints,International Journal of Metaheuristics, 2013 Vol.2, No.4,pp.415 - 433

2 S. Yassa, R. Chelouah, H. Kadima, B. Granado,Multi-Objective Approach for Energy-Aware WorkflowScheduling in Cloud Computing, The Scientific World Journal,vol. 2013, Article ID 350934, 13 pages, 2013.doi:10.1155/2013/350934

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 30 / 33

Page 31: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Publications(2/3)

• Conferences:

1 S. Yassa, R. Chelouah, H. Kadima, B. Granado, A GeneticAlgorithm Approach to QoS based Workflow Scheduling inCloud computing Environment, Proceeding of ICDSD’12International Conference en Distributed Systems and Decisions,ISSN 2335-1012, Oran-Algeria, November 21-22, 2012.

2 J. Sublime, S. Yassa, G-S. Jo, A genetic algorithm with theconcept of viral infections to solve hard constraints in workflowscheduling, In proceeding of: Korea Intelligent InformationSystem Society, At Seoul National University, Seoul, p95-100,2012

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 31 / 33

Page 32: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Publications(3/3)

• EURO Conferences:

1 S. Yassa, R. Chellouh, H. Kadima, B. Granado, A GeneticAlgorithm Approach to a Cloud Workflow Scheduling Problemwith Multi-QoS Requirements, 26th European Conference onOperational Research, Rome 1-4 july, 2013

2 S. Yassa, R. Chelouah, H. Kadima, B. Granado, A PSO-basedHeuristic for energy-aware scheduling of Workflow applicationson cloud computing, 25th European Conference onOperational Research, Lithuania 2012.

• workshops:

1 S. Yassa, H. Kadima, Ngociation et composition dynamiquedes services base sur le SLA dans un Cloud, Workshop surl’Evolution du principe de Rutilisation entre Composants,Services et Cloud Services(RCS2), juin 2011, Lille

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 32 / 33

Page 33: Optimisation de l'ordonnancement de workflows …cerin/VICHY2014/YASSA...ow Scheduling in Cloud computing Environment, Proceeding of ICDSD’12 International Conference en Distributed

Context & MotivationsProblem Description

Proposed SolutionsExperiments

Conclusion & PerspectivesEISTI

Thank you

Are there any

• Questions?

• Comments & Suggestions

Sonia Yassa Vichy 2014 QoS-based Workflow Scheduling in Cloud Computing 33 / 33