Data Mobility for Kubernetes Persistent VolumesXing Yang, Principal Architect, Huawei
@2000Xyang
Agenda
• Kubernetes Persistent Volumes and Snapshots
• Container Storage Interface (CSI)
• Why Data Mobility - Use Cases
• Policy Based Data Mobility using OpenSDS
• Multi-Cloud Data Control Architecture
• Demo
• OpenSDS Community
• OpenSDS Roadmap
• Getting Involved
Kubernetes Persistent Volumes and Snapshots
Kubernetes Persistent Volumes
• Persistent Volume• Persistent Volume
Claim• Storage Class• Dynamic and
Static Provisioning
Pod 1 Pod2Pod 1 Pod2
Storage
Container 1
Container 2
Container 1
Container 2
Volume 1 Volume 2
Container 1
Container 2
Container 1
Container 2
Volume 3 Volume 4
Kubernetes Node 1 Kubernetes Node 2
Kubernetes Volume Snapshots
• Volume Snapshot• Volume Snapshot Content• Volume Snapshot Class• Dynamic and Static
Provisioning• Supports CSI Drivers
User
Admin
PodContainer
Container Container
Container
StorageClass
PersistentVolumeClaim
Dynamic Provisioning
VolumeSnapshot
VolumeSnapshotClass
PersistentVolume VolumeSnapshotContent
PodContainer
Container Container
Container
iSCSI Ceph RBD
GCE Cinder
NFS Hostpath
FC
AWS
3rd Party Storage
Container Storage Interface (CSI)
Storage
CSI Plugin
K8S
CSI Functions
Operation based on Volume Lifecycle
Create Volume
Detach from node
Mount on node
Unmount from node
Attach to node
Delete Volume
ControllerCreateVolume
ControllerPublishVolume
NodePublishVolume
NodeUnpublishVolume
ControllerUnpublishVolume
ControllerDeleteVolume
Create Snapshot
Delete Snapshot
NodeStageVolume
NodeUnstageVolume
Create Volume from Snapshot
ControllerDeleteSnapshotControllerCreateSnapshot
Deploy CSI Plugin on Kubernetes
Source: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md
KubeController Manager
API Server
etcd
MasterNode
CSI Volume Driver Container
external-snapshotterexternal-provisionerexternal-attacher(1 or more sidecar containers)
EmptyDir Volume
gRPC (UDS)
StatefulSet (replica: 1)
CSI Volume Driver Container
driver-registrar sidecar container
Host /var/lib/kubelet
UDS
DaemonSet PodKubelet
/…/sockets
UDS
UDS
Mountpropagation
Why Data Mobility
Why do we need Data Mobility
• Data mobility allows data to be accessible any time, any where, on premise or in the cloud.
• Use cases for data mobility by OpenSDSmulti-cloud data control– High availability– Data analytics– Data lifecycle management– ……
Use Case: High Availability
• Make data available at different geographic location.
• Make data available and accessible all the time without downtime.
Use Case: Data Analytics
• Upload datasets to pubic cloud for data analytics.
• Utilize compute resources in public cloud to do CPU intensive work.
0
1
2
3
4
5
6
Category1
Category2
Category3
Category4
Series 1 Series 2 Series 3
Use Case: Data Lifecycle Management• Frequently accessed data
stored on-premise.• Data not accessed for a long
time can be stored in archival storage in public cloud.
• Data can be moved to different location depending on usage, age, access frequency.
• Data can be deleted automatically based on policy.
Block StorageOn-Premise
Object Storage
Migration Policy
SnapshotPolicy:
Topology
Archive Policy
RetentionPolicy
On-Premise -> Public Cloud Archive DeleteUpload Snapshot
Policy-basedData Mobilityusing OpenSDS
OpenSDS Projects – Sushi and Hotpot
SUSHI The Northbound Plug-ins ProjectCommon plug-ins to enable OpenSDS storage services for cloud and application frameworks
HOTPOTThe Storage Controller Project
Single control for block, file, and object services across storage on premise and in clouds
OpenSDS HotpotDiscover, Pool, Provision, Orchestrate
D O C K – S T O R A G E D R I V E R S
SwordfishManilaNative Cinder Multi-Cloud
C O N T R O L L E R - S E R V I C E A P I
Block File Object
Direct AttachedStorage
Software-Defined Storage
EnterpriseStorage
CloudStorage
CLOUD ENTERPRISE
Plug-inPlug-inPlug-inPlug-inPlug-inPlug-in Plug-in Plug-in
OpenSDS Project - Gelato
App
X
Local CloudStorage(Object)
Replicator XX
X
X
X
X
Gelato The Multi-Cloud ProjectPolicy based multi-cloud data control to enable data mobility across clouds
Profiles and StorageClass/VolumeSnapshotClass
Enterprise Storage
Create PVC &VolumeSnapshot
K8S calls OpenSDSCSI Plugin
OpenSDS CSI Plugin callsOpenSDS controller to provisionvolume & snapshot
OpenSDS chooses storagebackend to create volume (PV) and
snapshot (VolumeSnapshotContent)
CommodityStorage
Cloud Storage
User
AdminCSI PluginOpenSDSCreate profiles Create StorageClass &
VolumeSnapshotClass
Admin
CSI Plugin and Policy-based Data Mobility• Create snapshot using OpenSDS CSI
plugin• Upload snapshot based on snapshot
topology• Migrate snapshot based on migration
policyOpenSDS Controller
Dock
Driver
Block Storage
Snapshot Profile
Object Storage
CSI PluginOpenSDS
Migration Policy
Snapshot Class
Migration Policy
Snapshot Topology
Volume Snapshot Example
Upload SnapshotCut Snapshot
Create Snapshot
apiVersion: snapshot.storage.k8s.io/v1alpha1kind: VolumeSnapshotmetadata:
name: new-snapshot-demonamespace: demo-namespace
spec:snapshotClassName: csi-opensds-snapclasssource:
name: opensdspvckind: PersistentVolumeClaim
apiVersion: snapshot.storage.k8s.io/v1alpha1kind: VolumeSnapshotClassmetadata:
name: csi-opensds-snapclasssnapshotter: csi-opensdspluginparameters:
profile: upload-snapshot-to-cloud
Note: Profile can be uuid or name of the profile in OpenSDS.
Multi-Cloud Data Control Architecture
Multi-Cloud Data Control Architecture
S3 Service Backend Service Dataflow Service
Database
Backend Adapters
GUI API Gateway
Data Mover
Identity Service
Multi-Cloud Backend Service
Backend Manager
API Handler
Backend
DB
Keystone
create/update/delete/get backend
auth
update metadata
check bucket
Multi-Cloud S3 Service
S3 Controller
API Handler
Backend
Location Mapper
DB
Backend Adapter
Keystone
get locationsearch
create/update/delete/get object/bucket
auth
update metadata
Multi-Cloud Dataflow Service
Connector Manager
API Handler
Migration Data Mover
Policy Manager
Job Manager
Plan Manager
Other Data Mover …
DB
Scheduler
Apache Kafka
DataFlow Manager
Keystone
put jobs
pull jobs
run plan
create/delete/update/getpolicy
create/delete/update/getconnector
create/delete/update/getplan
get jobsrestart/cancel job
get plancreate jobsupdate plan
auth
update job
Multi-Cloud Dashboard
Multi-Cloud Dashboard – Register Backend
Multi-Cloud Dashboard - Migration
Demo
• Take snapshot using OpenSDS CSI plugin• Snapshot uploaded to AWS S3• Migrate data from AWS S3 to Huawei
Cloud object store
OpenSDS Roadmap
OpenSDS Community
Getting Involved
• Repos: https://github.com/opensds• Slack: https://opensds.slack.com• Mailing list: https://lists.opensds.io• Weekly meetings:
https://github.com/opensds/design-specs/blob/master/README.md#opensds-technical-meetings