x. liu, j. chen, z. wu, z. ni, d. yuan, y. yang, ccgrid10, 20-05-2010, melbourne, australia handling...
TRANSCRIPT
![Page 1: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/1.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Handling Recoverable Temporal Violations in Scientific Workflow Systems: A Workflow Rescheduling Based Strategy
Xiao Liu1, Jinjun Chen1, Zhangjun Wu2, Zhiwei Ni2, Dong Yuan1, Yun Yang1
1CS3, Swinburne University of Technology Melbourne, Australia
2Institute of Intelligent Management, Hefei University of Technology Hefei, China
![Page 2: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/2.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Outline
> Background
– Workflow Technology Group
– SwinDeW Family
– SwinGrid, SwinCloud
> Workflow Rescheduling
– Workflow Rescheduling
– Temporal Violations
– A Two Stage Workflow Local Rescheduling Strategy
– Evaluation
> Summary
2
![Page 3: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/3.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Workflow Technology Group Overview
> About Us: WT group is a part of CS3 (The Centre for Complex Software Systems and Services), a Tier-1 university research centre at Swinburne University of Technology. Our group conducts research into workflow technologies for complex software systems and services including peer-to-peer, grid, and cloud computing based e-science, e-business, transactional and inter-organisational workflows.
3
Leader:Prof Yun Yang
Visitors (7-8/09):Prof Lee OsterweilProf. Lori Clarke
Researchers:Dr Jinjun ChenNauman Saeed (PhD)Qiang He (PhD)Ke Liu (PhD)Xiao Liu (PhD)Dong Yuan (PhD)Zhangjun Wu (PhD - visitor)
Others:Prof Ryszard KowalczykProf Chengfei Liu
Dr Jun Yan (Wollongong)Prof Hai Jin (HUST)Prof Mingshu Li (ISCAS)Prof Qing Wang (ISCAS)Prof Zhiwei Ni (HFUT)Prof Jinpeng Huai (BUAA)
![Page 4: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/4.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
SwinDeW Family
SwinDeW – Swinburne Decentralised Workflow- foundation prototype based on p2p
> SwinDeW – past
> SwinDeW-A (for Agents) – ARC DP06
> SwinDeW-G (for Grid) – past
> SwinDeW-V (for Verification) – current (pending ARC DP)
> SwinDeW-E (for eScience) – current (pending ARC DP)
> SwinDeW-C (for cloud) – current (ARC LP)
> Others: SwinDeW-B / -S / -P / -G – past
4
![Page 5: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/5.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
SwinGrid to SwinCloud
5
Swinburne Computing Facilities
Astrophysics Supercomputer
VMware
Cloud Simulation Environment
Data Centres with Hadoop
· GT4· SuSE Linux
Swinburne CS3
…...
…...
· GT4· CentOS Linux
Swinburne ESR
…...
…...
· GT4· CentOS Linux
![Page 6: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/6.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Outline
> Background
– Workflow Technology Group
– SwinDeW Family
– SwinGrid, SwinCloud
> Workflow Rescheduling
– Workflow Rescheduling
– Temporal Violations
– A Two Stage Workflow Local Rescheduling Strategy
– Evaluation
> Summary
6
![Page 7: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/7.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Workflow Rescheduling
> A workflow scheduling is a process that maps the execution of inter-dependent workflow tasks on the distributed resources. It allocates suitable resources to workflow tasks so that the execution can be completed to satisfy objective functions imposed by users.
> Workflow rescheduling is a process to regenerate or modify the current scheduling plan, due to
– Changes of system environments: resource pool change, resource performance variance, resource break down, etc.
– Contract (constraint) violations: task execution failures, temporal violations, cost violations, etc.
– Many others: when the current scheduling plan cannot satisfy the pre-defined functional or non-functional objectives
![Page 8: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/8.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Workflow Rescheduling
> Workflow rescheduling plays an equally or even more important role as workflow scheduling in highly distributed and dynamic computing environments
– As a supplement function to workflow scheduling in the resource management component
– As a function in the exception handling component
> Example rescheduling strategy:
– Rescheduling by Stop and Restart
– Rescheduling by Processor Swapping
> Simple (little additional programming); high cost for resource reservation
![Page 9: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/9.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Temporal Violations
> Most scientific workflows are subjected to
– Local temporal constraints (milestones) for workflow segments
– Global temporal constraints (deadlines) for workflow instance
> Violations of local and global temporal constraints
– Detection: Temporal Checkpoint Selection (ICSE08, TOSEM10)
– Handling: Time deficits compensation (CCPE07, CCGrid10)
> Statistical recoverable temporal violations
– Probability based temporal consistency model (BPM08, ICSP09)
– Based on “3sigm” rule in statistics
– Details omitted, please refer the paper for details
![Page 10: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/10.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Outline
> Background
– Workflow Technology Group
– SwinDeW Family
– SwinGrid, SwinCloud
> Workflow Rescheduling
– Workflow Rescheduling
– Temporal Violations
– A Two Stage Workflow Local Rescheduling Strategy
– Evaluation
> Summary
10
![Page 11: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/11.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
A Two Stage Workflow Local Rescheduling Strategy
> For handling temporal violations
> Key objective: reduce or ideally remove the time deficit at the current checkpoint, i.e. to reduce the execution time of the subsequent activities after the checkpoint in the violated workflow segment as much as possible
> Requirement 1: fighting good balance between time deficit compensation and the completion time of other activities (workflow activities and general tasks, with or without temporal constraints) – from the overall makespan perspective
> Requirement 2: utilising available resources in the system rather than recruiting additional resources – from the overall cost perspective
11
![Page 12: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/12.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Integrated Task Resource List
12
![Page 13: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/13.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
13
Pseudo-code for An Abstract Strategy
![Page 14: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/14.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Evaluation
> Two example implementation of our strategy
– An ant colony optimisation based strategy
– A genetic algorithm based strategy
![Page 15: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/15.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Optimisation on Total Makespan
15
![Page 16: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/16.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Optimisation on Total Cost
16
![Page 17: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/17.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Compensation on Violated Workflow Segment
17
![Page 18: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/18.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Fitness Value
18
![Page 19: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/19.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
CPU Time
19
![Page 20: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/20.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Experiment Results on Temporal Violation Rates
20
![Page 21: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/21.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Outline
> Background
– Scientific Workflows
– Workflow Scheduling and Rescheduling
> Workflow Scheduling
– Classification
– Representative Scheduling Algorithms
> Workflow Rescheduling
– Classification
– A Two Stage Workflow Local Rescheduling Strategy
– Case Study 1:GA Based Rescheduling
– Case Study 2: ACO Based Rescheduling
– Comparison
> Summary
21
![Page 22: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/22.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
Summary
> Workflow Rescheduling – Exception Handling
> Exception Handling on Scientific Workflow Temporal Violations
– A Two Stage Workflow Local Rescheduling Strategy
• ACO, GA
– Automatic, Cost-Effective
> Future Work
– Data movement cost
– More scheduling algorithms
22
![Page 23: X. Liu, J. Chen, Z. Wu, Z. Ni, D. Yuan, Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia Handling Recoverable Temporal Violations in Scientific Workflow](https://reader034.vdocuments.net/reader034/viewer/2022051516/56649dc75503460f94abc62e/html5/thumbnails/23.jpg)
X. Liu , J. Chen , Z. Wu , Z. Ni , D. Yuan , Y. Yang, CCGrid10, 20-05-2010, Melbourne, Australia
The End – Thank You!
> Any questions or comments?
> Email: [email protected]
> Website: http://www.ict.swin.edu.au/personal/xliu/
> CS3: http://www.swinburne.edu.au/ict/research/cs3/
23