Download - S3 Server Hackathon Presented by S3 Server, a Scality Product, Seagate and Holberton School
Welcome to theS3 Server Hackathon!
Cloud Field Days #1 © Scality 20163
Scality RING: Automatize storage for Digital Business
The Scality RING is object-based software-defined storage for the cloud.
We run on standard x86 servers and create a giant pool of storage.
We protect the data and provide 100% reliable, high performance access for any capacity-driven application.
FILE OBJECT OPENSTACK
Cloud Field Days #1 © Scality 20164
Agenda for today
Presentation of the hackathon, judges and logistics
S3 Server tech Overview - https://github.com/scality/s3
Seagate Kinetic drives Overview
Teams and projects assignement!
The Judges!
The Prizes!
How are we judging?
We know the code will not be beautiful!
We’re looking for a functional prototype
It’s OK not to handle quirky corner cases!
How are we judging?
Innovation - We’re looking for creative solutions
Market potential - It has to solve a real problem!
Execution - A working demo ☺
Each team has:
A name,
A mentor,
A clear defined project that’s achievable,
Developers and a leader,
Each team will do a 5 minutes pitch + demo on Sunday
Logistics!
–––
Open source S3 Server!
S3 Adoption keeps rising
© Scality 201614
▪ Open source object storage serverhttps://github.com/scality/s3
▪ Written in Node.js▪ Single instance running in a docker
container▪ Uses docker volumes for persistent storage▪ Same code as Scality’s RING S3 interface
What is S3 Server?
© Scality 201615
▪ Developers can install and develop S3-based apps locally
▪ Enterprises can host a local test/dev environment to learn about object storage
▪ Enterprises can host a small, local object storage system in production
Scality S3 Server: From 0 to S3 in Under 5 Minutes
S3 Serve
r
S3 Serve
r
S3 Serve
r
Backup Application
S3
The teamSF & Paris
© Scality 2016
Scality S3 Server achitecture
Protocol StackBuckets, Objects, MPU (REST API)
AUTH BUCKETS DATA
LevelDB*
Scality RING v6
Kinetic IP drives
Docker Volume
* Persisted in Docker Volume
Scality RING v6Vault
JSON*
S3 API
S3 Server
Public Cloud
© Scality 201618
S3 server released under a Apache-2.0 on Github3.3K downloads on dockerhub!
18
Seagate KineticKinetic development and the basics
Kinetic On SMR (Shingled Magnetic Recording)
What is Kinetic
Kinetic is a drive architecture in which the drive is a key/value service with Ethernet connectivity
Put/Get/Delete API over TCP/SSL
https://www.openkinetic.org/
SMRData recording technology used in drives to increase storage density and overall storage capacity
https://en.wikipedia.org/wiki/Shingled_magnetic_recording
Why Kinetic on SMRIncrease storage capacity
Same Put/Get/Delete API (SMR is transparent to users/applications)
Kinetic Development OverviewKinetic Simulator
Functionally equivalent to Kinetic Drives.
https://github.com/Kinetic/kinetic-java
Build
git clone https://github.com/Kinetic/kinetic-java.git mvn clean package
Run (one or more instances) java -jar kinetic-simulator/target/kinetic-simulator-"Version"-SNAPSHOT-jar-with-dependencies.jar or./bin/startSimulator.sh
More Information Related to Buildhttps://github.com/Kinetic/kinetic-java/blob/master/README.md
Preferred IDE: Eclipse (Mars and up)http://www.eclipse.org/downloads/packages/eclipse-ide-java-developers/neon1a
Notes On Kinetic K/V KeysKeys are sorted lexicographically. e.g.
http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/primitives/UnsignedBytes.html#lexicographicalComparator%28%29
https://github.com/Kinetic/kinetic-java/blob/master/kinetic-simulator/src/main/java/com/seagate/kinetic/simulator/persist/leveldb/KineticComparator.java
Use sequential keys for better throughput (not required for later releases)
An example to generate sequential keys
https://github.com/Kinetic/kinetic-java/blob/master/kinetic-test/src/test/java/com/seagate/kinetic/KVGenerator.java
Hackathon Group Key Space Each group is recommended to use its own assigned key space (prefix) to avoid conflict for shared drives. For example, group 1 is assigned prefix of “00”, group 2 is assigned prefix of “01”., etc. Eg:
All keys in group 1 would be in the form of “000”, “001”, “002” ..., etc.
All keys in group 2 would be in the form of “010”, “011”, “012” ..., etc.
All keys in group 3 would be in the form of “020”, “021”, “022” ..., etc.
Scality/Kinetic Hackathon
Thank You!
Thank you!