desy cloud, the scientific data cloud - dcache · june 1, 2016 , frankfurt, patrick fuhrmann et al....

INDIGO DataCloud

DESY Cloud,The Scientific Data CloudManagedSharedStorageAtthe“ownCloud ConnectsBusiness”workshop

Dr.PatrickFuhrmannQuirin BuchholzTigranMkrtchyanPetervanderReestLusine Yakovleva

June1,2016,Frankfurt,PatrickFuhrmannetal. 2TheScientificDataCloud@ownCloud ConnectsBusiness

Content

• Storage@DESY?• Sync’n ShareatDESY

• Motivation• Requirements• Implementation• Setup

• RequirementsfromScienceCommunities.• dCache forDummies.• TheownCloud– dCache Hybridsystem• Summaryandoutlook.


Storage@DESY

• PetraIII[Tier0](2012…)• SynchrotronRadiation

• 14Beamlines• BeamlineGuestScientists

• 1PB/year– 5PB/year

• European[Tier0]XFEL (2017…)• 3.4Km(Linear)• 2017(Firstbeamline)

• BeamlineGuestScientists• 10– 100PB/year

• HERA[Tier0](1992– 2007)• Particleaccelerator(Proton– Electron)

• 6.3Km(Ring)• Somehundredscientists

• 5PBintotal

• LCG[WLCGTier2](2008,2009 …)• Particleaccelerator(Proton– Proton)• 26.7(Ring)

• About10.000scientist• 15PB/year

2020100PBytes

1992


MorestorageatDESY

•TheDESYdatamanagementteamhasquitesomeexperienceinmanaginghugeamountsofdata.

• Incollaborationwithother‘bigdata’sites,weareprovidingadatamanagementsystem‘dCache’,deployedat70sitesaroundtheworld.

• Seelater.•So,whyarewerunningownCloud ?


Motivation

• DESYhasnoexperienceinsophisticateddatasharing.• DatasharingwasdoneinthetraditionalwaywithACL’sand’group’directories

• However:YoungscientistsstarttheircareersatUniversitiesandLab’swithSync’n Shareintheirblood.(DropBoxGeneration).

• PublicITdepartments,foraverylongtime,didn’tregardSync’n Shareasbeingtheirproblemasmanycommercialsolutionswerearound.

• ItessentiallybecameanissueafterSnowden.• LegalRequirement:Datahadtobestored‘onsite’oratleastinGermany

• Consequence:CCneededtoprovideSync’n Sharelikemechanisms.


Requirements

• Finegrainedsharingoffilesanddirectorieswithindividualsandgroups.

• SharingviaintuitiveWeb2.0mechanisms(AppsorBrowser)• Sharingwith‘thepublic’withorwithoutpasswordprotection• Sharingofspacetouploaddata.(protected)• Expirationofshares• Automaticbidirectionalsynchronizationofdatabetweenmobiledevicesandcentralrepository.


TypicalApplication

Your Cloud SpaceSync

Sync

File up and download


StepstakenbyDESY• Evaluatedpossiblesolutionsin2013.• DecidedtogoforownCloud

• Providesmostofthefeaturesneeded.• OpenSource• WasinusebymanyinstitutesandUniversitiesinGermany• UsedbycolleaguesatSURFSara (Amsterdam)andCERN

• Evaluationshowed:• VerygoodSync’n Sharefeature set• Verygoodinplanningahead(roadmap)• Plansforcrosssitefederatedaccess(nowinplace).• Abitweakindatamanagement

• StartedprototypeinstallationatDESYbeginningof2014


WhatshouldtheDESYSetuplooklike?

(ActuallywilllooklikeinJuly)


TheInfrastructure

AuthenticationKerberos

UserManagementRegistryLDAP

Monitoring

LocalandWide AreaNetworkLoadBalancing Firewalls

Virtualization

Accounting 8 UnlimitedPersistentStorage


Infrastructure Integration

PostgresDB

OwnCloud

OwnCloudOwnCloud

OwnCloud

F5,LoadBalancer

AutomaticFailover


MoreIntegration

DESYKerberos

OwnCloud

8UnlimitedCentral

Storage

DESYLDAPDataLifeCycle

Engine


PoolNode

PoolNode

PoolNode

PoolNode

PoolNode

PoolNode

200TBytesRAID6

200TBytesRAID6

200TBytesRAID6

Horizontally ScalingBackend

OwnCloud OwnCloud OwnCloud OwnCloud

NFS4.1/pNFS

WebLoadBalancer(F5)


SomeStatistics

Filesin/outin7days10.000

70.000Filesin/outperhour

Users Total 490

Users Active 277

SpaceAvailable 567TBytes

SpaceUsed 2*30TBytes

Files 10Millions

CurrentDefaultQualityTwoReplicasondifferentstoragenodes.


Isthatsufficient forscientists?


TypicalWorkflow

Derived PublicationRaw

Sharing


DataCategories

1TB

10- 100TB

1– 100PB Raw

Derived

Publication

LHCDetectordataRawX-RayImagesBrainScansReconstructed(Ntuples)PurifiedImagesBrainMaps

Papers,Presentations,Histograms

Amount Category TypicalApplication


Whatdoweneedtosupport ‘scienceworkflows’?


MoreRequirements

• Storagemustbemanageable:DefinedQoS andDataLifecycle• DifferenttypeofdatamusthavedifferentQoS attached,regardingaccesslatency(performance)anddatadurability(howsafeismydata?)

• SpinningDiskforstreaming• SSDforfastrandomaccess• Tapeforarchive• Multiplecopiesindifferentlocationsondifferentmediaforlongtermdatapreservation

• MovingdatabetweendifferentQoS typeshastobeperformed• w/oserviceinterruption• transparentlytotheuser• w/ochangesinthenamespace


QualityofService

Raw

LongTermPreservation(LegalRequirement)

Derived

SSD

LowLatency(HPC,Analysis)

Publication

SSD

Fast,MultiStreamAccess


EvenmoreRequirements

• Differentaccessprotocolsfordifferentapplications• POSIXMountedFS(nfs4.1/pNFS) forfastanalysis• FTPdialects(gridFTP) forwideareatransferswithGLOBUS,WLCG-FTS• http/WebDAVmostlyforbrowserbasedapplications,visualization,..

• Differentauthenticationmechanismmustbeavailable.• Username/passwordforwebapplications• SAMLtosupporttraditionalIdP’s• OpenIDConnectforgoogle/facebook likeIdP’s• CertificatesforhttpsorGRIDapplications

• Differentcredentialsmustbemap-abletothesameidentity.


ScientificDataCloud

HighSpeedDataIngest

FastAnalysisNFS4.1/pNFS

WideAreaTransfers(Globus Online,FTS)byGridFTP

Sync’ing andSharingwith OwnCloud


Whatwouldthatlooklikefromtheuser’sperspective?


MyDESYXXLHomeQoS support

Patrick’shome


MyDESYXXLHomeProtocolSupport

MultiProtocolNFS4.1/pNFS

GridFTPWebDAVSRM

MyownCloud Home SyncShare

Web2.0ownCloud


Howdoweachievethosegoals?

ORChoosingdCache asthestoragebackendfor

ownCloud !

Thescientificdatacloud


SideTrack

What’sdCache ?


dCache inanutshell (cont.)

• Started2000’• Internationalcollaboration(DESY,FERMIlab,NDGF)• About10members:developers,deployment,support,management• Softwaredeployedatabout70sitesEurope,US,Asia,Russia• Largestdeploymentsintheorderof20PBytes ontapeanddisk.• Totalstoragecloseto200PBytes.• Geographicallylargestinstallationspans4countries.• LargelyfundedbyINDIGO-DataCloud,DESY,FERMIlab andNDGF

INDIGO DataCloud


dCache Design

MediaTransferEngineandPoolManagement dCache

Automaticand

ManualMedia

transition

Virtual file-systemnamespaceLayerProtocoland Authentication Engines

gridFTPNFS/pNFS httpWebDAV

SSDs

SpinningDisks

Tape, BlueRay…


NamespaceDesign

NameSpace PhysicalStorage

Disk

Tape

ExternalSystem

LocationManager

Name

Disk1

Disk2

Tape1


DesignConsequence

• Filesarestoredasobjectsonvariousdataback-ends• RandomDevices :Harddisk,SSD• RemovableMedia:Tape• Objectstores:CEPH

• Back-endscanbehighlydistributed(evenbeyondcountries).• TheFilenamespaceengineisindependentofthedatastorageitself.• Internalandexternalservicescanmovedataaroundw/oserviceinterruption.


dCache Featuressupporting ourideaofascientificdatacloud

• MultiProtocolSupport(TransferandAuthentication)• Transferprotocols:NFS/pNFS,http,WebDAV• MultiAuthenticationCredentialsupport(OpenIDConnect,Kerberos,passwd)

• SophisticatedDataManagement• MultiMediasupport(Tape,SpinningDisk,SSD,…)• Automaticandmanualmediatransitions• Addingandremovingdatanodesw/oserviceinterruption• Automaticreplicamanagement

• Enforcesn<x<mcopiesofdatafiles.• Externalstoragesupport(e.g.Tapesystems:TSM,HPSS,OSM,DMF)


Inparticular :TheQoS Interface


dCache QoS Interfaces

WebService

CDMIService

Cloud

dCache

QoSModule

RESTful


TheQoS WebInterface

DISK TAPE

Click,togetFilebackfromTape.


Puttingpiecestogether


TheDataPath

OwnCloud OwnCloud OwnCloud OwnCloud

NFS4.1/pNFS

WebLoadBalancer(F5)

SpinningDisks

SSD’s TAPE

dCache


FutureWorkTheNamespacePath

Namespace

NamespacedCache

SharingDB

ShareAPI

Namespace,Proxy


dCache – OwnCloud hybrid

• Datapathistheeasiestpart.Worksnicely.• Namespacesynchronizationis/wasverydifficult

• Importanttoletallprotocolsseesynchronizednamespace.• ownCloud didn’texpecttheunderlyingstoragesystemtochangenamespacetree.• Manuallytriggeredsynchronizationtooktoolong.• OwnCloud 9providesfirstattemptforanAPIforexternalnamespace.

• Exposing‘shares’toexternalcomponentnotyetinownCloud.• ImportanttoallowallprotocolstouseownCloud-definedshares.• Prerequisites:

• ownCloud :needsAPItoexpose‘shares’• dCache :needstohavea‘share’objectimplemented.


ownCloud andQoS

I/O(NFS)

ownCloud GUIWeb

dCacheNamespaceAPI

ShareAPI

QoSPluggin

(ServerSideApp)

QoSModule

RESTServices


Summary

• AnOwnCloud - dCache Hybridisaperfectsystemforprovidingmanagedsharedstoragetoscientists.

• Sync’n ShareisprovidedbyownCloud.• AccessprotocolsandAuthenticationMechanismsusedinscienceareprovidedbydCache.

• Unlimitedstoragespaces(viaremovablemedia,e.g.tape)• QualityofServicesupport

• automaticandmanualmediatransitions• Automaticreplicamanagementresultinginhighavailabilityanddatadurability.

• Reduceddowntimesduetotransparentdatamigration.


Outlook

• ThecurrentversionoftheownCloud-dCacheHybridsatisfiestheneedfor

• Sync’n Share• Highlyscalableandmanageableback-endstorage

• Forafullintegration• Thename-spacesofthetwosystemsneedtobesynchronized(OC9)• TheownCloud ‘shares’needtobeexposedtohavethemvisibleinallprotocols(nfs,gridFTP,…)

• WeneedtoprovideanownCloudpluggin(serversideapp)tomakethedCacheQoSstoragetypesvisibleinownCloud.


TheEND

furtherreadingwww.dCache.org

desy cloud, the scientific data cloud - dcache · june 1, 2016 , frankfurt, patrick fuhrmann et al....

Documents