context-aware cloud computing for hep applications

Context-awarecloudcomputingforHEPapplications

RandallSobieInstituteofParticlePhysics(Canada)

UniversityofVictoria

RandallSobieIPP/Victoria 1

Context-awaresystemsinmobilecommunication

RandallSobieIPP/Victoria 2

Howcanweusetheinformation(context)ofa

usertotheirbenefit?

LocationDirectionTimeHealthSocial

Context-awaresystemsinmobilecommunication

RandallSobieIPP/Victoria 3

Theinformationcanmeeton-demandrequestorbeproactivelysentto

theuser

DirectionsFoodandlodging

FinancialCommercial

Health

RandallSobieIPP/Victoria 4

EntityAperson,place,or

objectthatisrelevanttotheirinteraction

Context-broker

ContextInformationusedto

characterizethesituationofanentity

Context-brokerCombinesreal-timecontext

ActionContext-triggeraction

RandallSobieIPP/Victoria 5

Entity

Context-broker

Context

Action

Content-aggregator

Content-aggregatorStorehistoricalcontextthatcanbeusedbythecontext-brokertoinitiate

anaction

Context-broker

RandallSobieIPP/Victoria 6

EntityISGCPhysicist

Context-broker

ContextHungry

DinnertimeCentralTaipei

ActionListofrestaurants

Context-broker

RandallSobieIPP/Victoria 7

EntityISGCPhysicist

Context-broker

ContextHungry

DinnertimeCentralTaipei

ActionListoffastfood

Content-aggregator

Content-aggregatorGraduatestudent

Context-broker

Context-awarecloudcomputingsystem

RandallSobieIPP/Victoria 8

EntitiesWorkloadsystems

CloudsVMs

VM-provisionerBatchjobscheduler

Canacontext-awaredesignmakeiteasier,moreefficienttorunHEPapplications?

UtilizedynamicandopportunisticresourcesAutomaticallyself-configurecloudsandvirtualmachinesLocateoptimalsoftwarecachesanddatarepositories

Fault-detectionanderrorcorrectionIntelligentdesignandself-learning

CloudcomputinginHEP

RandallSobieIPP/Victoria 9

Opportunistic

DedicatedVirtual

cluster

CloudcomputinginHEPistypicallyproviding5-20%oftheprocessingofcurrentprojects

“Dedicated”clouds(OwnedbyHEP)

“Opportunistic”clouds(privateandcommercial)

HEPuseofcloudsforbatchworkloads

RandallSobieIPP/Victoria 10

IaaSAPI

ApplicationProgramInterface

JobScheduler

VMlife-cyclemanager

A“VMlife-cyclemanager”managesVMsbasedonthejobqueueCloudhasnoapplication-specificrequirements

Userorworkload

manager

Multipleremoteclouds

RandallSobieIPP/Victoria 11

JobScheduler

VMlife-cyclemanager

Userorworkload

manager

IaaSAPI

IaaSAPI

IaaSAPI

Workloadmanagerhasnoknowledgeoftheresources

Distributedbatchcloudsystem

RandallSobieIPP/Victoria 12

Dedicatedandopportunisticresources(ATLASandBelleII)

17%ofBelleIIcomputingin2015usedclouds

Productionuseofcloudsfor3-4yearswithgradualincreaseinutilizationTechnologyisstillyoungandrapidlyevolving

Integratingnewtechnologiesintoaproductionsystemischallenging

Evolutiontoacontext-awarearchitecture

RandallSobieIPP/Victoria 13

Softwarecachesandrepositories

Monitoringandcommunication

services

BatchjobVM

provisioningservices

Datafederationservices

Otherservices Contextbroker

Goalisasystemthatisdynamic,automated,intelligent,error-correctingandabletoexploitanycloudwillingtoprovideresources

Contextinformationsystem(s)keepingtrackofreal-timeinformationStateofclouds,services,squid-caches,datafederations

Usethecontextinformationtoconfigure,operateandmonitor

Identifyandmonitorclouds–matchjobstoresourcesVMsselfconfigureandmonitor,locatethesoftwareanddataRecognizechangesanderrors–takecorrectiveactions

Context-collection

RandallSobieIPP/Victoria 14

Wealreadycollectdataandlogsonservices,clouds,userjob,VMsReal-time(messagepassing)communicationbetweensomeservices

Numberofrunningjobs

Snapshotofcurrentsystem

Forexample,monitoringpageforoneoftheATLAScloudsystemsinCanada#activeclouds#jobs(1c/8C)#VMs#jobslots

VMprovisioning

RandallSobieIPP/Victoria 15

JobScheduler

IaaSAPI

IaaSAPI

IaaSAPI

VMprovisioningserviceCloudScheduler

Userjobrequirements

Joblist Resourceavailability

Entity

Context

Context-broker

VMprovisioning

RandallSobieIPP/Victoria 16

JobScheduler

IaaSAPI

IaaSAPI

IaaSAPI

VMprovisioningserviceCloudScheduler

JSsendsjobtoVM Cloud-BbootsaVMfortheuserjobandregisterswithJS

Entity

Action

Context-broker

Findsthecloudthathasidleresourcesandmeetstheuser

requirements

Softwaredeliveryservice

RandallSobieIPP/Victoria 17

ListofsquidsLocation

LoadVirtualmachinelocation

Entity

Context

Context-broker

SquidHTTPcache

SquidHTTPcache

SquidHTTPcache

SquidHTTPcache

CVMFSServer

CERNVMfilesystem

ShoalShoalAgroupofsquids

Softwaredeliveryservice

RandallSobieIPP/Victoria 18

VMisgiventhenearestHTTPSquidcache(OSandAppsoftware)

Entity

Action

Context-broker

SquidHTTPcache

SquidHTTPcache

SquidHTTPcache

SquidHTTPcache

CVMFSServer

CERNVMfilesystem

ShoalShoalAgroupofsquids

Real-timeCloudandVMstatus

RandallSobieIPP/Victoria 19

Softwarecachesandrepositories

Monitoringandcommunication

services

BatchjobVM

provisioningservices

Datafederationservices

Otherservices Contextbroker

Wewouldlikereal-timemonitoringinformationoftheremotecloudandindividualVM

Automatedfault-recoveryresponse

Commonissues:PeriodicallyacloudgoesofflineApplicationjobisstalledVMisstalled–notresponsivetotheVM-provisionserviceVMislost–notregisteredwiththeVM-provisionserviceLocalnetworkissuesinthecloudVMevictionsduetopricefluctuationsincommercialcosts

Possibleactions:Cloudisautomaticallydisabled

VMde-registeredfromHTCondor

Datafederationservice

RandallSobieIPP/Victoria 20

Storageelement

Storageelement

Storageelement

Storageelement

Datafederationservice

DataLocationNetwork

Requestfordataset

Datafederationservice

RandallSobieIPP/Victoria 21

Storageelement

Storageelement

Storageelement

Storageelement

Datafederationservice

ListoflocationsRetrievefromonesiteorstreamfrom

multiplesites

Datafederationservice

RandallSobieIPP/Victoria 22

Storageelement

Storageelement

Storageelement

Storageelement

Datafederationservice

MultiplecopiesdistributedaroundtheworldHEP-specific:FAX

HTTP/WebDav:Dynafed

Otherideas–futureplans

RandallSobieIPP/Victoria 23

IaaSAPI

ApplicationProgramInterface

TypicallycloudresourcesinOpenStackareassignedtoprojectsusing“tenants”Effortstoshareresourcesbetweentenantsinanautomatedmannertofullyutilizetheresources(e.g.sharingofATLASandBelleIIresources)

ConnectcloudresourcestotheLHCONEnetwork?(e.g.Softwaredefinednetworks–SDN)

Commercialcloudlinkstoresearchnetworks

Summary

•  UseofcloudcomputinginHEPisgrowing

–  UsingdedicatedandopportunisticresourcestoprovideanincreasingfractionoftheneedsofHEPexperiments

•  Context-awarecloudcomputingwillenableustoscaleouruseof

distributedresourcesthatisrequiredbyHEPprojects

–  UtilizetechnologiesdevelopedinsideandoutsideHEPtosimplify

management,expandfunctionalityandeaseofuse

•  Valuabletootherresearchcommunities

RandallSobieIPP/Victoria 24