context-aware cloud computing for hep applications
TRANSCRIPT
Context-awarecloudcomputingforHEPapplications
RandallSobieInstituteofParticlePhysics(Canada)
UniversityofVictoria
RandallSobieIPP/Victoria 1
Context-awaresystemsinmobilecommunication
RandallSobieIPP/Victoria 2
Howcanweusetheinformation(context)ofa
usertotheirbenefit?
LocationDirectionTimeHealthSocial
Context-awaresystemsinmobilecommunication
RandallSobieIPP/Victoria 3
Theinformationcanmeeton-demandrequestorbeproactivelysentto
theuser
DirectionsFoodandlodging
FinancialCommercial
Health
RandallSobieIPP/Victoria 4
EntityAperson,place,or
objectthatisrelevanttotheirinteraction
Context-broker
ContextInformationusedto
characterizethesituationofanentity
Context-brokerCombinesreal-timecontext
ActionContext-triggeraction
RandallSobieIPP/Victoria 5
Entity
Context-broker
Context
Action
Content-aggregator
Content-aggregatorStorehistoricalcontextthatcanbeusedbythecontext-brokertoinitiate
anaction
Context-broker
RandallSobieIPP/Victoria 6
EntityISGCPhysicist
Context-broker
ContextHungry
DinnertimeCentralTaipei
ActionListofrestaurants
Context-broker
RandallSobieIPP/Victoria 7
EntityISGCPhysicist
Context-broker
ContextHungry
DinnertimeCentralTaipei
ActionListoffastfood
Content-aggregator
Content-aggregatorGraduatestudent
Context-broker
Context-awarecloudcomputingsystem
RandallSobieIPP/Victoria 8
EntitiesWorkloadsystems
CloudsVMs
VM-provisionerBatchjobscheduler
Canacontext-awaredesignmakeiteasier,moreefficienttorunHEPapplications?
UtilizedynamicandopportunisticresourcesAutomaticallyself-configurecloudsandvirtualmachinesLocateoptimalsoftwarecachesanddatarepositories
Fault-detectionanderrorcorrectionIntelligentdesignandself-learning
CloudcomputinginHEP
RandallSobieIPP/Victoria 9
Opportunistic
DedicatedVirtual
cluster
CloudcomputinginHEPistypicallyproviding5-20%oftheprocessingofcurrentprojects
“Dedicated”clouds(OwnedbyHEP)
“Opportunistic”clouds(privateandcommercial)
HEPuseofcloudsforbatchworkloads
RandallSobieIPP/Victoria 10
IaaSAPI
ApplicationProgramInterface
JobScheduler
VMlife-cyclemanager
A“VMlife-cyclemanager”managesVMsbasedonthejobqueueCloudhasnoapplication-specificrequirements
Userorworkload
manager
Multipleremoteclouds
RandallSobieIPP/Victoria 11
JobScheduler
VMlife-cyclemanager
Userorworkload
manager
IaaSAPI
IaaSAPI
IaaSAPI
Workloadmanagerhasnoknowledgeoftheresources
Distributedbatchcloudsystem
RandallSobieIPP/Victoria 12
Dedicatedandopportunisticresources(ATLASandBelleII)
17%ofBelleIIcomputingin2015usedclouds
Productionuseofcloudsfor3-4yearswithgradualincreaseinutilizationTechnologyisstillyoungandrapidlyevolving
Integratingnewtechnologiesintoaproductionsystemischallenging
Evolutiontoacontext-awarearchitecture
RandallSobieIPP/Victoria 13
Softwarecachesandrepositories
Monitoringandcommunication
services
BatchjobVM
provisioningservices
Datafederationservices
Otherservices Contextbroker
Goalisasystemthatisdynamic,automated,intelligent,error-correctingandabletoexploitanycloudwillingtoprovideresources
Contextinformationsystem(s)keepingtrackofreal-timeinformationStateofclouds,services,squid-caches,datafederations
Usethecontextinformationtoconfigure,operateandmonitor
Identifyandmonitorclouds–matchjobstoresourcesVMsselfconfigureandmonitor,locatethesoftwareanddataRecognizechangesanderrors–takecorrectiveactions
Context-collection
RandallSobieIPP/Victoria 14
Wealreadycollectdataandlogsonservices,clouds,userjob,VMsReal-time(messagepassing)communicationbetweensomeservices
Numberofrunningjobs
Snapshotofcurrentsystem
Forexample,monitoringpageforoneoftheATLAScloudsystemsinCanada#activeclouds#jobs(1c/8C)#VMs#jobslots
VMprovisioning
RandallSobieIPP/Victoria 15
JobScheduler
IaaSAPI
IaaSAPI
IaaSAPI
VMprovisioningserviceCloudScheduler
Userjobrequirements
Joblist Resourceavailability
Entity
Context
Context-broker
VMprovisioning
RandallSobieIPP/Victoria 16
JobScheduler
IaaSAPI
IaaSAPI
IaaSAPI
VMprovisioningserviceCloudScheduler
JSsendsjobtoVM Cloud-BbootsaVMfortheuserjobandregisterswithJS
Entity
Action
Context-broker
Findsthecloudthathasidleresourcesandmeetstheuser
requirements
Softwaredeliveryservice
RandallSobieIPP/Victoria 17
ListofsquidsLocation
LoadVirtualmachinelocation
Entity
Context
Context-broker
SquidHTTPcache
SquidHTTPcache
SquidHTTPcache
SquidHTTPcache
CVMFSServer
CERNVMfilesystem
ShoalShoalAgroupofsquids
Softwaredeliveryservice
RandallSobieIPP/Victoria 18
VMisgiventhenearestHTTPSquidcache(OSandAppsoftware)
Entity
Action
Context-broker
SquidHTTPcache
SquidHTTPcache
SquidHTTPcache
SquidHTTPcache
CVMFSServer
CERNVMfilesystem
ShoalShoalAgroupofsquids
Real-timeCloudandVMstatus
RandallSobieIPP/Victoria 19
Softwarecachesandrepositories
Monitoringandcommunication
services
BatchjobVM
provisioningservices
Datafederationservices
Otherservices Contextbroker
Wewouldlikereal-timemonitoringinformationoftheremotecloudandindividualVM
Automatedfault-recoveryresponse
Commonissues:PeriodicallyacloudgoesofflineApplicationjobisstalledVMisstalled–notresponsivetotheVM-provisionserviceVMislost–notregisteredwiththeVM-provisionserviceLocalnetworkissuesinthecloudVMevictionsduetopricefluctuationsincommercialcosts
Possibleactions:Cloudisautomaticallydisabled
VMde-registeredfromHTCondor
Datafederationservice
RandallSobieIPP/Victoria 20
Storageelement
Storageelement
Storageelement
Storageelement
Datafederationservice
DataLocationNetwork
Requestfordataset
Datafederationservice
RandallSobieIPP/Victoria 21
Storageelement
Storageelement
Storageelement
Storageelement
Datafederationservice
ListoflocationsRetrievefromonesiteorstreamfrom
multiplesites
Datafederationservice
RandallSobieIPP/Victoria 22
Storageelement
Storageelement
Storageelement
Storageelement
Datafederationservice
MultiplecopiesdistributedaroundtheworldHEP-specific:FAX
HTTP/WebDav:Dynafed
Otherideas–futureplans
RandallSobieIPP/Victoria 23
IaaSAPI
ApplicationProgramInterface
TypicallycloudresourcesinOpenStackareassignedtoprojectsusing“tenants”Effortstoshareresourcesbetweentenantsinanautomatedmannertofullyutilizetheresources(e.g.sharingofATLASandBelleIIresources)
ConnectcloudresourcestotheLHCONEnetwork?(e.g.Softwaredefinednetworks–SDN)
Commercialcloudlinkstoresearchnetworks
Summary
• UseofcloudcomputinginHEPisgrowing
– UsingdedicatedandopportunisticresourcestoprovideanincreasingfractionoftheneedsofHEPexperiments
• Context-awarecloudcomputingwillenableustoscaleouruseof
distributedresourcesthatisrequiredbyHEPprojects
– UtilizetechnologiesdevelopedinsideandoutsideHEPtosimplify
management,expandfunctionalityandeaseofuse
• Valuabletootherresearchcommunities
RandallSobieIPP/Victoria 24