sebasen goasguen –sebgoa@clemson - fermilabcd-docdb.fnal.gov/0040/004050/001/cloud-fermi.pdf ·...

August 20th, Fermi Lab 1

Sebas&enGoasguen–[email protected]

SchoolofCompu,ng

ClemsonUniversity,Clemson,SC

Scien,ficAssociateatCERN

Summer2009andSummer2010


Outline

•  CloudBasics•  BuildingaCloudProvider

– Lxcloud@CERN

•  VOCsandClouds– ResearchdoneatClemson


WhatisCloudCompu&ng?


Afewreferences

"Above the clouds: A Berkeley view of cloud computing"http://berkeleyclouds.blogspot.com/

"A break in the clouds: towards a cloud definition"L.M Vaquero et al. SIGCOMM computer communication review, 2008. http://portal.acm.org/citation.cfm?id=1496100

"An EGEE Comparative Study - Grid cloud comparative study"M-Elian Begin, 2009


OntheHypecurve

• NowprobablyatthetopoftheHype–Oct09


Trendy…

• Source:hOp://www.google.com/trends


Cloudforma&on

• SlideadaptedfromRichWolski,UCSB


WhiteHouseisgoingtotheCloud• Reducecosts…SeeApps.gov


DOEandNASAtoo(Checknovacc.org)


EverythingismovingtotheCloud…StayonEarththough!

• hOp://contactdubai.com/tag/saas‐soTware‐or‐storage‐as‐a‐service


An“Old”idea:OSI/AnatomyoftheGrid/Windowsarchitectures…


WhatistheCloud?The*aaS

•  SaaS–SoVwareasaService‐•  PaaS–PlaYormasaService‐

•  Iaas–InfrastructureasaService–•  Servicecomposi,onatalllayersofdistributedsystem.Buildsasystemofsystem

•  SoTwareandhardwarereuse

•  Tendencyforthe*aaS‐itusbutthesethreearethemainones


SoVwareasaService


Skyisthelimit…

•  Phoneapps…FermiVoice?


PlaYormasaService


InfrastructureasaService/Comingofageofvirtualiza&on


WhatistheCloud?The*aaS•  SaaS–SoVwareasaService–

– EasyAccesstohostedapplica,onsoverthenetwork.MostlikelyusingyourBrowser

– APItotheseapplica,ons•  PaaS–PlaYormasaService–

– Environmenttodeploynewapplica,ons– Restrictedcapabili,esoffered– APItothispla]ormandaccesstoSaaSAPI

•  Iaas–InfrastructureasaService–– AccesstoHardwareresources– APItomakeresourcealloca,onrequests


KeyFeatures•  Youdon’tknowwhat’sbehindbutitworks

– Transparency•  YouPaywhatyouuse

– U,litypricing•  Yougetwhatyouaskfor(On‐demand)

– ReadthefineprintsoftheSLAs…•  Itscalesifyouneedmore

– Howfardoesitscale?– Doesn’tthismeantheunderlyingresourcesareunderu,lized?


Whynow?Evolu&onoftheMashupRevolu&onthanks

toanAPI“explosion”


Whynow?•  BigInternetcompaniesfacedalotofdatatoanalyze:weblogs…

•  Developedinhouse:Newfilesystem(Hadoop),newanalysisframework(Map‐Reduce)

•  Massiveamountofresourcesallacrosstheplanet:>500,000coresforGoogle?

•  Higherneedstoconsolidate:virtualiza,on,energycosts.

•  Newdevices:iPhone/G1•  Atrulyinter‐connectedplanet


Afewinteres&ngthings…tos&rthepot•  Industryisleading.Isacademiabehind?

•  Whocaresaboutstandards?(>20bodiesworkingoncloudstandards…)

•  Weshouldswitchparadigmandrewriteapplica,onsoncetheyare6monthsold.


Outline


– Lxcloud@CERN(Incollabora&onwithUlrichSchwickerath,EwanRoche,BelmiroMoreiraandRomainWartel)



IaaSlevel•  Forconsolida,ngservices

– UsedinITforawhilenow– FermiGridservicesrunninginXenVMs

•  Forofferingon‐demandservices– E.gVOBoxes,replacehardwarerequest

•  Forvirtualizinglargescaleservices– Clusterson‐demand

•  Virtualiza,onisakeyenablerforIaaS


BatchVirtualiza,on

  RunbatchjobswithinVirtualMachines   BeOerapplica,onenvironment

  Custommadebyuser   Increasedsecurity   BeOercontrolonresourcesharing

  Mul,‐coreapps   Increasedflexibilityontheadminside

  CanrunapreferredOSonthemetal

Whyvirtualizing“Batch”?


BatchVirtualiza,on

Type1:Runmyjobs(inyourVM)

Type2:RunmyjobsinmyVM

Howtovirtualize“Batch”…smoothly?


Type3:Givememyinfrastructurei.eaVMorabatchofVMs

Movingtothecloud:

August 20th, Fermi Lab 27 3/23

Deployment Models Innovation in Cloud Computing Architectures


Maincomponents/characteris&csSetofHypervisors•  Physicalmachineswithavirtualmachinemonitor•  XenorKVM...orHyper‐V...orVMwareESx...VMprovisioningsystem•  OpenNebula•  Nimbus•  Eucalyptus•  Pla]ormISF•  oreventradi,onalschedulerslikePBS/Maui.Imagedistribu&onmechanism•  Sharedfilesystem(e.gNFS,AFS,PVFS,Lustre...)•  Copyimages(e.gscp,wget,BiOorent)Networking•  Private/Publicbridged•  NAT


ThoughtsforOSG…tos&rthepotagain•  Sitesneedtohavehypervisors,that’sastar,ngpoint.Withoutit/themtherewon’tbeOSGclouds.

•  WhatVMM/HypervisortheyusedoesnotmaOer…butmyguessisthat80%willuseKVM

•  WhatprovisioningsystemtheyuseisamaOeroflocaltechnicalsetup,tasteandrela,onships

•  Sitescandothisnow•  Thehardproblemisintheimagetransferandtrust…SeeHEPiXvirtualiza,onworkinggroup


CERN'sLXCLOUDarchitecture

•  ImagerepositorywithGoldennodes.

•  VMinstancesnotquaOormanagedhavefinitelife,me

•  SpecificIP/MACsarepinnedtohypervisors

•  Currentlytes,ngtwoprovisioningsystem:OpennebulaandPla]ormISF.


ProvisioningsystemOpenNebulaandPla]ormISFarecurrentlybeingevaluated.ResultsshowninthistalkwereobtainedwithOpenNebula.

OpenNebulaoutoftheUniversityCompultenseofMadrid•  C/C++corewithRubydriversandcommandlineinterface•  MysqlandSqlitebackends•  Usesshascommunica,onbetweenfrontendandhosts•  XML‐RPCAPI•  •  SupportforLVMcontributedbyCERN•  EnablesHybridclouds(i.einstan,a,ononremotecloudproviders)

•  ImplementssubsetofEC2interfaceaswellasupcomingOCCIinterfaceforPubliccloudinterface.


ComparisonwithSimilarTechnologiesOpenNebula - Architecture, Current Status & Roadmap

Platform ISF VMware Vsphere Eucalyptus Nimbus OpenNebula

Virtualization Management VMware, Xen VMware Xen, KVM Xen Xen, KVM,

VMware

Virtual Network Management Yes Yes No Yes Yes

Image Management Yes Yes Yes Yes Yes

Service Contextualization No No No Yes Yes

Scheduling Yes Yes No No Yes

Administration Interface Yes Yes No No Yes

Hybrid Cloud Computing No No No No Yes

Cloud Interfaces No vCloud EC2 WSRF, EC2 EC2 QueryOGF OCCI

vCloud

Flexibility and Extensibility Yes No Yes Yes Yes

Open Source No No GPL Apache Apache


CERN'sLXCLOUDdetails

•  AlogscpandbiOorrentimagedistribu,onhasbeenimplemented

•  Hypervisorsrunu,li,estodetectwhatVMtheyareallowedtorunandwhichimagestheyneedtodownload

•  OpenNebulatriggersinstan,a,onviassh

•  InstancesbasedonLVMsnapshots


ImageDistribu&on

Push:•  Sequen,alSCP•  logarithmicSCP(scp‐wave)•  hOp://code.google.com/p/scp‐wave/

Pull:•  wgetviaanhOpbasedrepository(locally)•  BiOorrent(RomainWartel,BelmiroMoreira@CERN)

SharedFS•  NFS•  PVFS,Lustre...


Imagedistribu&onresults(thxtoBelmiro)


Guidingtheprovisioning

•  Definepoliciestocomposethebatchfarm•  Automatetheprovisioningofthevirtualmachinessuchthatthepoliciesareenforced.

•  e.gInspectthejobqueueanddeducethebestcomposi,onofthebatchfarm.IntermsofSMPVMs,OS...

•  AsizerisusedtomonitorthepoolofVMinstancesandevaluatethepolicies.

•  Currentlyonlyonepolicy:"KeepthepoolfullwiththepropersharesofVMtypes"

•  SeeICAC2010andCCGRID2009papers


AutonomicProvisioningResults


EarlyResultsofsizer


JoiningtheBatchsystem...Acontextualiza=onproblem...

CONTEXT = [ vmid = "$VMID", TTL = "3", AFS = "off", files = "/opt/vmimage/init.sh /opt/vmimage/etchosts /opt/vmimage/etcsysconfigifcfg /opt/vmimage/id_rsa.pub /o pt/vmimage/lsfcontext.conf /opt/vmimage/etcsysconfignetwork", target = "xvdb" ]

• FilesandvariablesarestoredinaISOcreatedonthefly.

• StartupscriptmountsthisISOandrunscontextualiza,onscript.

• VMsaresetupasdynamichostsintheLSFpool.


ScalabilityTests...7,500slotsinLSFviaOpennebula


Tes&ngLSFscalability


IaaSatClemson•  Thereallyeasyway:

– KVMonaregularHPCcluster

– NATnetworking(everyVMgetsitsownNAT)

– BaseimageonNFSserver– KVMsnapshotmodecreatestemporarydiskinscratch,diskdiscardedonceinstanceisshutdown

– SubmitVMsasPBSjobs

IMAGE=/home/sebgoa/kvm/star5.img

export TMPDIR=/local_scratch

kvm -hda $IMAGE -net nic,model=e1000 -net user -m 1280 -snapshot -nographic;


IaaSatClemson

•  But…– NosharedFSbetweenVMs

– LookslikeeachVMhasthesameIP

– Can’tuseregularjobmanagementsystemstorunjobsinthoseVMs(needglidein/proxylikesolu,on)

•  ThissetuphasbeenoneofthekeydriversforourdevelopmentofKestrel:AnXMPPbasedjobmanagementsystem


Kestrel

•  AjobmanagementframeworkusingtheXMPPprotocol

•  Startedasastudentproject

•  UsesInstantMessagingconceptsofno,fica,ons

•  Prac,calinadversenetworkcondi,ons

hop://wiki.github.com/legastero/Kestrel/hops://twiki.grid.iu.edu/bin/view/CampusGrids/InstallingKestrel


Boo&ngVMsisextremelyfast(20VMs/sec)


STARSuccesswithClemsonIaaSandKestrel

•  “Buttosimulatetheequivalentsampleof12.2BillionMonte‐Carloeventswith~10MillionacceptedbyeventtriggeringaTerfulleventreconstruc,on,wewouldhavetaken3yearsatBNLon50machinesThisMonte‐Carloeventgenera,onwouldessen,allynothavebeendone.Withtheresourcesfromcloud,wetook3‐4weeks.”–JeromeLauretBNL.


Conclusions•  TheCloudisherelet’shopeitgetssunny•  APIexplosionopensuppossibili,es•  FocusingonIaaSlayers,LXCLOUDandClemson’sclustershavebeendeveloped/enhancedtoprovisionVMs.

•  GreatscalabilitywithOpenNebula•  KVMshowsgreatpromiseespeciallywiththesnapshotmode

•  PerformancewillgetevenbeOer• MayneedspecializedjobmanagementsystemstomakeuseofCloudsacrossmul,‐site


ThankstoNSF,DOEandOSGThankstoLanceStout,MikeMurphy,

MichaelFenn,LintonAbrahamandalltheotherstudents…

ThankstoCERNandtheIT/PES‐PSgroupThankstoJeromeLauret,MaohewWalker

Ques&ons?:[email protected]://cirg.cs.clemson.edu


Outline


– Lxcloud@CERN(Incollabora,onwithUlrichSchwickerath,EwanRoche,BelmiroMoreiraandRomainWartel)



VOC:VirtualOrganiza&onCluster(JGC+FGCSpapers)


WhyVOCsakaClouds?•  Observa,onthatwhatpeoplewantisresourceswiththeirownOS/Appsandcentralscheduling:Pilotjobframeworks.

•  AcloudisaclusteroverWAN

•  Thereforethereisaneedfor– Awaytorequest/startthenodes– Awaytocreateavirtualnetwork– Awaytorunjobsinthem

•  VerysimilartoglideinWMSbutthepilotsasktostartVMs


Mul&‐SiteOverlay(ICAC2010)


VOCImplementa&on

•  Mul,pleconfigura,ons:– Type1:SharedheadnodeonPhysicalcluster,VOisunawareofVOC(e.gLXCLOUD)

– Type2:VOprovidesvirtualheadnodesonmul,plegridsites.

– Type3:VOusesanoverlaynetworkwithasingleheadnode(e.gSTAR).


Type1:Implementa&on

•  KVMvs.Xenforeaseofuse

•  NormalClusteru,li,es/techniques

•  NFSshare•  AndPVFSsetup

•  KVMoffersasnapshotmodethatgivesusabilitytouseasingleimagefile.Writesaretemporary


Load‐DrivenProvisioning(CCGRID09)•  DynamicProvisioningisdoneviatheuseofaWatchdogontheVOCheadnode

•  WatchdogmonitorsincomingjobsontheOSGgatekeeper(Condorjobmanagerisused)

•  Whenjobsareinthelocalschedulerqueue,thewatchdogstartsaVMonaphysicalhost(sta,cmappingbetweenhostandguestcurrently).XML‐RPCsystem

•  WhenVMstarts,CondorinsidetheVMstartsandadver,zesitspresencetothecentralmanager‐>Jobsrun.


ExperimentalResults•  EngageVOonOSG•  SiteClemson‐BirdnestonOSGProduc,on

•  Clustersizerespondstoload,Simula,onResultsconfirm(PendingIPDPSpaper,simulator:simVOCavailableathOp://cirg.cs.clemson.edu/soTware/simvoc)


From: ACAT 2010, February 22-27th Jaipur/India

Engage VO on OSG

sebasen goasguen –sebgoa@clemson - fermilabcd-docdb.fnal.gov/0040/004050/001/cloud-fermi.pdf ·...

Documents