Cloud Computing + Workflows
Post on 23-Feb-2016
DESCRIPTIONCloud Computing + Workflows. Anushri Khandekar. Cloud Computing. Delivering applications or services in on-demand environment Hundreds of thousands of users / applications Systems should be fast, secure and available Intelligent infrastructure: Transparency Scalability Monitoring - PowerPoint PPT Presentation
Cloud Computing + WorkflowsAnushri Khandekar
Delivering applications or services in on-demand environmentHundreds of thousands of users / applicationsSystems should be fast, secure and availableIntelligent infrastructure:TransparencyScalabilityMonitoringSecurityAll services and associated data
Operational aspect of a work procedure: how tasks are structured, who performs them, what their relative order is, how they are synchronized, how information flows to support the tasks and how tasks are being tracked.
Workflow ManagementAn activity is a discrete step in a business process (workflow). Activities range from calling a remote service to perform a task, e.g. calculating taxes, performing currency conversions, looking up inventory, to custom-defined services.Activities are orchestrated together in a workflow in BizTalk using XOML (eXtensible Object Markup Language).Other languages BPEL, ebXML, XPDL etc.
Workflows in CloudMicrosoft allows hosting of Biztalk activities in a cloud at biztalk labs.Developers integrate those cloud hosted activities into a BizTalk workflow (orchestration) by calling them as they would any other web-based service or hosted activity.Service orchestration business process is modeled using workflowsInvokes Internet Service Bus and perform HTTP requestLanguage used XOMLMain task First create a workflow instance and start it
Actual Implementation of services obscuredAnother version of virtualizationTransparent load-balancing and application deliverySolution to be automated and integrated in workflow processExample:A service running with a single server, more users join in hence additional servers required, transparency allows integration without interrupting the service running or reconfiguration.
Scale up and build mega data centersNot transparent Need configuration or re-architecting Potential of interrupting services is hugeAbility to transparently scale the service infrastructure and the solutionOn-demand, real time scalingControl node provides dynamic application scalabilityIntegration with virtualization solution or orchestration with workflow process to manage provisioning
Control node intelligent monitoring capabilitiesServer overwhelming or application performance affected by network conditions behavior outside accepted normsMore than knowing when a service in trouble what action should be taken Example application responding slowly, adjust application requests add more server if requiredDetect and participate in the provisioning of new instance
Capacity ManagementFrom buckets to riversConstrained set of resources predict peak usage and have in-house data centre to manage themUnlimited computing power with cloud How IT departments properly manage this river?Constraint on new modelNot upper limit of computing power but speed at which new services can be provisioned and put into productionScaling up means:Initiate new system, transfer data, connect existing system, test combined system, manage complete life cycle
Capacity ManagementTraditional life cycle stages:Modeling, provisioning, monitoring, maintaining, and modifying.Important here Maintaining & ModifyingElastic means provisioning and de-provisioningIs it right time to add an IT asset or get rid of an asset?Economic benefits rely on when to stop using an assetUtilize the cloud for additional capacity when it is apparent your own data centre can't handle the load and it is cost-prohibitive to invest in additional servers and infrastructure to increase capacity
Efficient management of workflows in a cloud environment to allow fast scaling up and scaling downStoring scalability/ compressibility options for every node in the workflowInput events and output events of every node in workflowMechanism to integrate new scaled model of web service in original cloud workflow
Workflow management important heavy workflow of traditional waterfall approaches with smallest detail will slow down the use of cloud computingSeparate main workflow from details of mechanism required to scale any activity nodeHave efficient way of storing this information
Workflow ManagementWorkflow MainHas the cloud structure with each web service as an activity nodeWorkflow ShadowHas sub-workflows for other options for each activity nodesWorkflows Online or Offline.Online running and executing at a particular time Offline workflows in passive state waiting for an event to trigger them
Parameters DescriptionActivity Node NameDescriptionTypeStateConstraintsInput EventOutput EventScalability OptionsCompressibility Options/* Unique Activity node Name*//* Description of Activity *//* Service or Application etc*//* Online, Offline, or Needs change*//* Time, Execution Cost *//*Event to trigger the activity node *//*Event triggered by activity node *//*Scalability activities as a workflow */ (When and How)/*Compressibility activities as a workflow*/(When and How)
Scalability OptionsConsidering transparency, two ways to scale a workflowScale an activity nodeAddition of new activity nodeMore tricky, dynamic, according to environmentScale an activity nodeWhen? store criteriaExample, for a web server if load increases above a threshold, expandHow? again as a workflowExample, store all the steps to be done in order to expand, configure and connect the node back to original workflow
Cloudbursting vs Bursting the CloudCloudbursting is to allow the cloud to act as overflow resources in the event your own infrastructure becomes overloadedCritical tasks (revenue generating) in own datacentre
Bursting in the cloud is applied to resources such as servers, application servers, application delivery systems, and other infrastructure required to provide on-demand computing environments
Bursting the cloudAutomate the cloud's data centreRequires more than simple workflow systemson-demand control and management over all devices in the delivery chainfrom the storage to the application and web servers to the load-balancers and acceleration offerings that deliver the applications to end-usersData centre orchestration many moving parts and pieces be coordinated in order to perform a highly complex set of tasks
Hadoop As a Service
Automated installation and provisioningResearch Questions:How to support multi-tenancy with QoS differentiationHow to optimize workflows across users with fluctuating capacity requirementsKey features: On-demand creationDynamic resource flexing
Differentiated Hadoop services
Problem:More important jobs should preempt less important jobsTime critical jobs need to meet deadlinesTest jobs need no stringent QoS guaranteesHow to get users to truthfully reveal their resource requirements?
Differentiated Hadoop services
ApproachMarket-based resource allocator, TycoonContinuous bidding (of spending rates) for resource capacityProportional allocationAllocation materialized as VMUsers can evaluate and select providers based on cost/benefit metrics (best value for money)Gives incentive to users to be judicial about capacity requests and time to submit
Economic workflow optimization
Assumption: Not all subtasks need maximum capacity at all times Approach: Automatically rescale the capacity as needed to optimize the cost/benefit ratio of the workflow as a whole Opportunity:Application scalability profile not perfectly linear
Optimization strategiesNode PriorityP: Some nodes more performance critical than others S: Boost spending on critical nodes (e.g. master funding boost)Workflow Priority: P: Some workflows more performance critical than others (although they look the same to the system)S: Declare relative priority of workflows and split budget accordinglyJob Priority:P: Some stages of a workflow are more i/o intensive, others more cpu intensiveS: Boost resource spending during resource-intense stages of workflowBottleneck Mitigation:P: During map/reduce synch up some nodes may be bottlenecks S: Redistribute funds to active bottlenecks
Optimization strategiesBest Response:P: When other users place competing bids, optimal configuration/allocation might change S: Find game theoretical best response bids continuously to maximize utilityRisk:P: Some users are more risk averse than others (can tolerate less fluctuations)S: Bid on nodes based on predicted guarantee to deliver a QoS level
Managing ResourcesIncludes clear policies on who to admit how to arbitrate among competing requests what resource capacity may be requested over what time framesIsolated DatacentreReset, reboot, power up, power down, get statusBias towards large and short experimentsSite coordination required, e.g. accounting
XOML Original Cloud ActivitiesCloudHttpSendCloudHttpReceiveCloudIfElseCloudSequenceActivity node details should be stored with thisCloudServiceBusSendCloudDelayCloudWhile
Citrix Cloud CentreXenServer Cloud Edition a complete, cloud-ready virtual infrastructure NetScaler to load balance, speed access to backend VMs and dynamically provision workloads. "There's more to providing [cloud computing] than simply providing a flat virtual infrastructure. You want to have workflows, you want SLAs, you want to be able to automate and move things around, and that's essentially what Citrix is bringing to the table -- a full suite of tools to do all of that." James Staten Citrix WANScaler and Citrix Workflow StudioSingle Automated Cohesive system
Workflow management matters because much of the benefits of cloud computing comes from the speed and ease with which IT resources can be created and put into production.
Thank you !!! Questions ???