dependable storage and computing using multiple cloud providers

Download Dependable Storage and Computing using Multiple Cloud Providers

Post on 23-Jan-2017

68 views

Category:

Software

3 download

Embed Size (px)

TRANSCRIPT

ReMIOP: Projeto e Implementao de um Mecanismo de Difuso Confivel no CORBA

Dependable Storage and Computing using Multiple Cloud ProvidersAlysson Bessani

www.di.fc.ul.pt/~bessani

International Industry-Academia Workshop onCloud Reliability and ResilienceBerlin, 7-8 November 2016

#

1

Abstract & BioDespite the widespread success of public cloud services, issues such as outages, data loss, and vendor lock-in have prevented many organizations toembrace the cloud. An intuitive way to avoid these problems would be to replicate data and services in different clouds, avoiding thus the need tocompletely trust any single cloud provider. In this talk Ill discuss the benefits and challenges of this approach, with special focus on the practicalinsights gained from building several multi-cloud storage systems.

Alysson Bessaniis an Associate Professor ofthe Faculty of Sciences of the University of Lisboa, Portugal, and a member of LaSIGE research unit. He holds aPhDin Electrical Engineering fromFederal UniversityofSanta Catarina, Brazil (2006), was avisiting professor at Carnegie Mellow University(2010), and a visiting researcher at MicrosoftResearch Cambridge (2014). Alysson co-authored more than 100 papers, participated in several FP7 and H2020 projects and is coordinating the DiSIEM H2020 project. His current interestslie indistributed systems, Byzantine fault tolerance,cloud storage, blockchain design and applications, and security monitoring. More information about him can be found athttp://www.di.fc.ul.pt/~bessani.

#

Cloud Sec. & Dep.Secure and dependable services are a necessary condition for the long-term existence of a cloud providerClients need to trust providersProviders need to justify this trust

To stay in the market, a provider needs to invest on Sec. & Dep., however

#

UnavailabilityNumber of nines99,9% = 8,76 hours99,99% = 53 minutes

#

http://www.networkworld.com/article/2866950/cloud-computing/which-cloud-providers-had-the-best-uptime-last-year.htmlhttp://www.networkworld.com/article/3020235/cloud-computing/and-the-cloud-provider-with-the-best-uptime-in-2015-is.html

1

Unavailability

#

Data Loss

201120152016

#

Privacy Issues

#

#

Important!To the best of my knowledge, there is no work on theexisting technical literature saying that using the cloudis less secure than having everything on premisesCloud services work pretty well but they are not perfect!

#

Our viewThe best way to use the cloud in a dependable and secure way is to not rely on a single providerDont base your service on a single provider, i.e., an application works if the provider where it is running is correctInstead, consider distributed trust: an application is correct if at most f out-of n services/providers are faulty

#

Why a Cloud of Clouds?Survive datacenter and cloud outagesAvoid vendor lock-inAchieve better read performanceTolerance to data corruptionBugsMalicious insidersAttacks and intrusions

#

Cloud-of-clouds ServiceTwo fundamental characteristicsNo modification on existing cloud servicesNo collaboration between providers

#

Cloud-of-Clouds Storage

#

Cloud Storage Diversity24+ cloud object storage servicesLocations: 68Model: Standard - ArchivalCost per GB/month stored: $0,001 - $0,13Cost per GB downloaded: $0,01 - $0,25Availability SLA: undefined, 99,9%-100%Access control: inexistent, URLs, , ACLs,Standards compliance: 0-33+

#

FundamentalsLets consider the problem of implementing a cloud-of-clouds object storage tolerating one cloud failureAssume you cannot know for sure if a unresponsive cloud is faulty or not

#

Writing Data

One needs to write at least two copies of the data!What if one of the clouds take too much to acknowledge the write?Use three clouds, and wait for at least two acks to complete the write

#

Reading Data

How many clouds you need to access for reading?One cloud: might access an outdated/empty cloudTwo clouds: always access the last complete write

writequorumreadquorum

#

Reading Data II

What if data can be modified or corrupted?

writequorumreadquorum

#

Reading Data II

To solve this we need more clouds and bigger quorums

writequorumreadquorum

#

DepSky: Dependable Cloud-of-Clouds Object Storage

[EuroSys11, ACM Trans. on Storage 2013]Availability, Integrity and Confidentialitydespite the failure of up to f clouds

#

Challenges for implementing Updatable ObjectsHow to implement an efficient replication protocol using only passive storage nodes?

How to make it affordable?

readwrite

#

Cloud ACloud BCloud CCloud DDDDDqwjdasjkhdahsdqwjdasjkhdahsdqwjdasjkhdahsdqwjdasjkhdahsdDepSky WriteWRITEDATADACKDDDDWRITEMETADATAqwjdasjkhdahsdACKqwjdasjkhdahsdqwjdasjkhdahsdqwjdasjkhdahsdqwjdasjkhdahsd

#

Cloud ACloud BCloud CCloud DDDDDDepSky ReadREADDATADDATADDDDREADMETADATAqwjdasjkhdahsdqwjdasjkhdahsdqwjdasjkhdahsdqwjdasjkhdahsdMETADATAqwjdasjkhdahsdData will be fetched from other clouds if needed. highest version number

#

Cloud ACloud BCloud CCloud DDepSky Confidentiality and Storage EfficiencyDataLimitations:Data is accessible by cloud providersRequires n|Data| storage spaceDataDataDataData

#

Cloud ACloud BCloud CCloud DDepSky Confidentiality and Storage EfficiencyS1S2S3S4shareKgeneratedkeyDatadisperseF1F2F3F4F1S1F2S2F3S3F4S4encrypt

Inverse process for reading from f+1 shares/fragments.

#

Consistency ProportionalityThe consistency provided by DepSky is the same as the base storage cloudsIf the weakest consistency cloud provides eventual consistency, DepSky provides eventual consistencyIf the weakest consistency cloud provides read your writes, DepSky provides read your writesIf the weakest consistency cloud provides regular storage, DepSky provides regular storageThis notion may be useful for other systems

#

Practical ConsiderationsRead/write latency accessing a single cloudStorage costs roughly 50% higherDepSky is a Java programming libraryhttp://cloud-of-clouds.github.io/depskyIt does not support concurrent writers to the same object without using locksRecent extensions for multi-writer storage (2016)http://github.com/cloud-of-clouds/mwmr-registersHow to build a complete system around it?

#

CoordinationService

SCFS: Shared Cloud-backed File System

Storage clouds

Cloud Storage

CacheCacheCache

LockServiceAccessControlMetadataComputingcloudsSCFSAgentSCFSAgentSCFSAgent[USENIX ATC 2014]

#

SCFS Consistency on Close

fd = open(x,);read(fd,);write(fd,);fsync(fd);write(fd,);close(fd);

CoordinationService

Cloud(-of-Clouds)

PersistentStorage

Main Memory

READWRITE

Local StorageIf x is cached, it is only validated1212

#

SCFS can use different backendsi.e., different cloud storage and a coordination service plugin

Operation: blocking, non-blocking and non-sharing

SCFS Backends

#

The Cloud-of-Clouds BackendDoes not require trust on any single cloud providerSCFS works correctly as long as less than a third of the providers misbehave

SCFSAgentDepSpace/BFT-SMaRt[EuroSys08, DSN14]DepSky CoC Storage[ACM ToS13]Cache

#

Sharing Latency: SCFS vs DropBox

Amazon S3Google StorageRackspace FilesWindows Azure BlobDATADATA?$#!%|

DATADATASCFSDropbox

#

Financial Evaluation

+50%

writeread

#

Practical ConsiderationsSCFS improves the transparency of cloud-of-clouds(-backed) storage It is implemented as a Linux FUSE FS:http://cloud-of-clouds.github.io/SCFSLimitations:Does not work well with big filesRequire computing instances to run the coordination service (e.g., Zookeeper replicas)Can we do better?

#

Private CloudPrivate CloudPrivate CloudCharon Cloud-backed File System

Charon

Charon

Charon

+ big data storage and sharing+multiple storage locations+ serverless design

#

Charon Building BlocksOne-Shot RegisterCan be written only onceUsed for storing file contentsSingle-Writer Multi-Reader (SWMR) RegisterCan be updated, but not concurrentlyUsed for storing file system metadataLease ObjectProvide lock/unlock operationsUsed for avoiding write-write conflicts in the system

#

Charon Architecture

Charon

Charon

TLS ChannelCache

Site 1Site 2Cache

C

A

Public Cloud APublic Cloud B

D

B

ABd1

Dd2Cmetadata

#

Avoiding write-write Conflicts without External Coordination

f-tolerant Byzantine leaseleasereleaserenewenqueue, dequeue, peekAllput, get, del, casatomic transactions

leasereleaserenew

Queue

leasereleaserenew

Queue

leasereleaserenew

DynamoDB

leasereleaserenew

Data Store

f+1clouds

#

Personal and Shared Namespaces

#

Confidentiality & Storage-EfficiencyCloud ACloud BCloud CCloud DS1S2S3secret sharingKrandomly generated keyCompressed Chunkerasure codeF1F2F3F1S1F2S2F3S3encrypt

Big FileChunk 1Chunk Ncompress

Optional

#

Processing Writes

#

FS Microbenchmarks

Data-intensive operations (MB/s)Metadata-intensive operations (ops/s)

#

Upload/Download Latency

0.6 GB/min1 GB/min

relatedwork

relatedwork

#

Cloud-of-clouds as a Service

Janus: User-defined

Recommended

View more >