exoscale: pithos: your personal s3 object store on cassandra
TRANSCRIPT
![Page 2: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/2.jpg)
@PYRCTO at Exoscale, Swiss Cloud Hosting.Open source developer: pithos, cyanite, riemann, collectd.
![Page 3: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/3.jpg)
AIM OF THIS TALKPresenting object storageShow-casing efficient uses of object storagePresenting pithosFeedback on usage
![Page 4: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/4.jpg)
OUTLINEObject Storage 1016 things you should do with S3Pithos, your personal Object StorePithos in production
![Page 5: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/5.jpg)
OBJECT STORAGE 101
![Page 6: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/6.jpg)
THE ELEVATOR PITCHObject Storage is a storage architecture that
manages data as objects
Wikipedia
![Page 7: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/7.jpg)
INCEPTIONAsset and content storage for large hosting platforms.Livejournal's MogileFS.A shift in how we perceive distributed storage.
![Page 8: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/8.jpg)
ESSENTIAL PROPERTIESNo POSIX guaranteesNo atomicityEventual consistencyPushes some responsibility back to the application.
![Page 9: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/9.jpg)
THE OBJECT STORAGE LANDSCAPEMostly hosted solutions:
AWS S3Rackspace Cloud FilesDreamObjectsExoscale SOS
No real API standardisationAWS S3 is the de-facto standard
![Page 10: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/10.jpg)
THE ON-PREMISE OBJECT STORAGE LANDSCAPESome vendor-backed solutions:
EMC AtmosScalityCloudian
SwiftCephRiak CSPithos
![Page 11: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/11.jpg)
A TYPICAL OBJECT STORE REQUEST# curl -X PUT -d @file.txt https://mybucket.myprovider.com/some-file.txt# curl https://mybucket.myprovider.com/some-file.txt
![Page 12: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/12.jpg)
S3 TERMINOLOGYRegion: Determines where objects will be stored.Storage Class: Storage properties for objects.Bucket: A named container for objects.Object: A file.
![Page 13: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/13.jpg)
THE S3 APIA global bucket namespaceArtificial hierarchy supportAuthentication and Authorization through ACLsMultipart uploadsCORS support & Form based uploadsEventual consistency
![Page 14: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/14.jpg)
A GLOBAL BUCKET NAMESPACEA single consistent namespace for buckets:
Across tenants.There is only one highlander bucket.
A bucket is located within a region.
![Page 15: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/15.jpg)
HIERACHY SUPPORTListing requests may supply a delimiter and prefix.Emulates directories when keys contain slashes.
![Page 16: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/16.jpg)
HIERARCHY SUPPORTGET /?delimiter=/ HTTP/1.1Host: mybucket.service.uriDate: <date>Authorization: AWS <key>:<signature>
![Page 17: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/17.jpg)
HIERARCHY SUPPORT<?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>batman</Name> <Prefix></Prefix> <MaxKeys>100</MaxKeys> <Delimiter>/</Delimiter> <IsTruncated>false</IsTruncated> <Contents> <Key>sample.txt</Key> <LastModified>2014-10-17T12:35:10.423Z</LastModified> <ETag>"a4b7923f7b2df9bc96fb263978c8bc40"</ETag> <Size>1603</Size> <Owner> <ID>[email protected]</ID> <DisplayName>[email protected]</DisplayName> </Owner> <StorageClass>Standard</StorageClass> </Contents></ListBucketResult>
![Page 18: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/18.jpg)
AUTHENTICATION & AUTHORIZATION THROUGH ACLSSimple canned ACLs allow common settings.
e.g: public.An XML syntax is also available.
![Page 19: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/19.jpg)
MULTIPART UPLOADSAllows uploading several chunks of files.User-controlled re-aggregation step.Limits the impact of upload failures for large files.
![Page 20: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/20.jpg)
CORS SUPPORT AND FORM-BASED UPLOADSWeb interaction without any backend components.CORS setup through an XML configuration syntax.Form based uploads through pre-signed requests.
![Page 21: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/21.jpg)
EVENTUAL CONSISTENCYAn easy sell at Cassandra SummitPossible delay between PUT and GET availability.Checksums avoid massive inconsistencies.
![Page 22: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/22.jpg)
6 THINGS TO DO WITH S3
![Page 23: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/23.jpg)
12-FACTOR APP SUPPORT FOR PERSISTENCEEliminates the need for NFSEases interaction with PaaS type platforms
http://12factor.net/
![Page 24: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/24.jpg)
STATIC CONTENT HOSTINGPerfect for hosting CSS, JS and other static assetsSimply requires setting a bucket's ACL to public
![Page 25: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/25.jpg)
FORM BASED UPLOADSPre-signed requestsRequests encapsulate a policyNo proxying to the S3 service requiredGreat for supporting user generated content
![Page 26: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/26.jpg)
ARTIFACT STORAGESupported in MavenSupported in Docker RegistrySupported in AptSupported in Mesos fetcher
![Page 27: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/27.jpg)
BACKUPSGreat Open-Source options like duplicity.Commercial storage gateway support.Some home NAS-type products support S3 as well.
![Page 28: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/28.jpg)
CLIENT-SIDE ENCRYPTIONGPG encryption support.Guarantees full data ownership, even when leveraging third-party providers.Don't lose your keys!
![Page 29: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/29.jpg)
PITHOS, YOUR PERSONAL OBJECT-STORE
![Page 30: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/30.jpg)
FROM THE WEBSITEPithos is a daemon which provides an S3-compatible frontend for storing files in a
Cassandra cluster.
![Page 31: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/31.jpg)
WHY ?Provide your own S3-compatible service (that's us!)Restricted from using hosted object-storage services.Willingness to fully own availability.
![Page 32: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/32.jpg)
PITHOS ESSENTIAL PROPERTIESExtensive S3 API coverage.Fully Stateless.Multi-region support.Fully Cassandra-backed.Extensible.Open-Source.
![Page 33: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/33.jpg)
MISC.Runs on the JVM.Written in Clojure.Small codebase (~ 5300 LoC).Can run an embedded cassandra for tests purposes.
![Page 34: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/34.jpg)
PITHOS ARCHITECTUREA daemon built out of 5 isolated and pluggable components.
![Page 35: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/35.jpg)
PITHOS ARCHITECTUREKeystoreBucketstoreMetastoreBlobstoreReporter
![Page 36: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/36.jpg)
OVERALL CONCEPT
![Page 37: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/37.jpg)
THE KEYSTOREAuthentication & Authorization handled outside of pithos.Only component which doesn't rely on Cassandra by default.Default implementation relies on the pithos configuration file.Maps an API key to a credentials.Example alternative implementation in the documentation.
![Page 38: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/38.jpg)
THE KEYSTORE{ "tenant": "tenant name", "secret": "secret key", "memberof": ["group1", "group2"]}
![Page 39: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/39.jpg)
THE BUCKETSTOREStores essential bucket properties
Bucket tenant.Region and storage-class where bucket is located.Optional CORS properties.
![Page 40: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/40.jpg)
THE BUCKETSTOREBucket ownership is transactional.Cassandra is not the best suited for this task.The lightweight transaction features help.
![Page 41: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/41.jpg)
THE BUCKETSTORE{ "bucket": "batman", "created": "2012-01-01 01:30:00", "tenant": "[email protected]", "region": "ch-dk-2", "acl": "...", "cors": "..."}
![Page 42: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/42.jpg)
THE METASTOREStores all object details.References an inode an version in the bucketstore.Using the path as a key in a wide colum ensures keys aresorted.
![Page 43: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/43.jpg)
THE METASTORE{ "bucket": "test", "object": "file.txt", "inode": "4e682d3d-28fa-4ea6-aa28-282c2757f31b", "version": "c97894cd-e2cd-46d5-a217-1add544e88a4", "atime": "2012-01-01 01:30:00", "size": 1024, "checksum": "d41d8cd98f00b204e9800998ecf8427e", "storageclass": "standard", "acl": "...", "metadata": { }}
![Page 44: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/44.jpg)
THE BLOBSTOREStores data.Inodes are lists of blocks.Blocks are lists of chunks.Chunks contain small (128k) chunks of the file.
![Page 45: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/45.jpg)
THE BLOBSTORENot what Cassandra was meant for.Works suprisingly well.
![Page 46: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/46.jpg)
THE REPORTEREmits useful usage information.Good basis for building billing extensions.
![Page 47: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/47.jpg)
CONFIGURATIONA single configuration file to configure all aspects
Logging & server options.Keystore, bucketstore, metastore and blobstore.Each can have its own details / cassandra cluster.
![Page 48: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/48.jpg)
CONFIGURATIONservice: host: "0.0.0.0" port: 8080logging: level: info console: true overrides: io.pithos: debugoptions: service-uri: s3.example.com default-region: myregion
![Page 49: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/49.jpg)
CONFIGURATIONkeystore: keys: AKIAIOSFODNN7EXAMPLE: tenant: [email protected] secret: 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'bucketstore: default-region: myregion cluster: "localhost" keyspace: storageregions: myregion: metastore: cluster: "localhost" keyspace: storage storage-classes: standard: cluster: "localhost" keyspace: storage max-chunk: "128k" max-block-chunk: 1024
![Page 50: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/50.jpg)
AREAS OF IMPROVEMENTV4 Signatures.Overall S3 API coverage.Overall S3 Client coverage.Promoting Cassandra compact storage.Simple web interface.More contributors and users!
![Page 51: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/51.jpg)
V4 SIGNATURESV4 type signatures are still not supported in pithos and are item
number 1 on the todo-list.
![Page 52: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/52.jpg)
OVERALL S3 API COVERAGEThe S3 API is byzantine and corner cases are poorlydocumented.Still missing some useful bits (versioning, bucket policies,session tokens).
![Page 53: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/53.jpg)
OVERALL S3 CLIENT COVERAGESome clients are very sensitive with regard to API behavior.The essentials work.Glitches are quickly fixed when caught.
![Page 54: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/54.jpg)
PROMOTING CASSANDRA COMPACT STORAGEWITH COMPACT STORAGE gives great benefits.Not yet promoted or automatically converged on startup.
![Page 55: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/55.jpg)
SIMPLE WEB INTERFACEA simple JavaScript SPA would be nice.
![Page 56: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/56.jpg)
PITHOS IN PRODUCTION
![Page 57: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/57.jpg)
A WORD OF WARNINGRunning an object-store is not necessarily for the faint of heart.
![Page 58: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/58.jpg)
HOW WE USE ITNo multi-datacenter clusters.Dedicated metadata cluster.Dedicated "blobstore" clusters.
![Page 59: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/59.jpg)
ELSEWHEREFew known installations (in the 10s).Always rather large.Always used where cassandra previously existed.
![Page 60: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/60.jpg)
MAINTENANCE (PITHOS)A few cases generate orphan inodes and must be prunedmanually.Internal tooling used for this, should eventually be released.Rather worry-free
![Page 61: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/61.jpg)
MAINTENANCE (CASSANDRA)The usual applies
Schedule regular repairs of your clustersFollow releases
Best supported version: 2.1.xQuorum is satisfactory in terms of performance.
![Page 62: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/62.jpg)
SCALINGPithos is stateless.Colocate cassandra and pithos daemons.Split blobstore and metastore keyspaces into separateclusters.Split Data/Proxy nodes is worth investigating for hugedeployments.Haproxy to distribute queries to pithos instances.
![Page 63: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/63.jpg)
PARTING WORDSTry it out! (There's an all-in-one version)Get involved
Docs need proof-reading, additions.Some issues need to be tackled.
![Page 64: Exoscale: Pithos: your personal S3 object store on cassandra](https://reader033.vdocuments.net/reader033/viewer/2022042506/587c19711a28abb5068b4deb/html5/thumbnails/64.jpg)
THANKS !Pithos owes a lot to:
Max Penet (@mpenet) for the great alia & jet librariesDatastax for the awesome cassandra java-driverIts contributorsApache Cassandra obviously
@pyr