Data Mobility for Kubernetes Persistent VolumesXing Yang, Principal Architect, Huawei

@2000Xyang

Agenda

• Kubernetes Persistent Volumes and Snapshots

• Container Storage Interface (CSI)

• Why Data Mobility - Use Cases

• Policy Based Data Mobility using OpenSDS

• Multi-Cloud Data Control Architecture

• Demo

• OpenSDS Community

• OpenSDS Roadmap

• Getting Involved

Kubernetes Persistent Volumes and Snapshots

Kubernetes Persistent Volumes

• Persistent Volume• Persistent Volume

Claim• Storage Class• Dynamic and

Static Provisioning

Pod 1 Pod2Pod 1 Pod2

Storage

Container 1

Container 2

Container 1

Container 2

Volume 1 Volume 2

Container 1

Container 2

Container 1

Container 2

Volume 3 Volume 4

Kubernetes Node 1 Kubernetes Node 2

Kubernetes Volume Snapshots

• Volume Snapshot• Volume Snapshot Content• Volume Snapshot Class• Dynamic and Static

Provisioning• Supports CSI Drivers

User

Admin

PodContainer

Container Container

Container

StorageClass

PersistentVolumeClaim

Dynamic Provisioning

VolumeSnapshot

VolumeSnapshotClass

PersistentVolume VolumeSnapshotContent

PodContainer

Container Container

Container

iSCSI Ceph RBD

GCE Cinder

NFS Hostpath

FC

AWS

3rd Party Storage

Container Storage Interface (CSI)

Storage

CSI Plugin

K8S

CSI Functions

Operation based on Volume Lifecycle

Create Volume

Detach from node

Mount on node

Unmount from node

Attach to node

Delete Volume

ControllerCreateVolume

ControllerPublishVolume

NodePublishVolume

NodeUnpublishVolume

ControllerUnpublishVolume

ControllerDeleteVolume

Create Snapshot

Delete Snapshot

NodeStageVolume

NodeUnstageVolume

Create Volume from Snapshot

ControllerDeleteSnapshotControllerCreateSnapshot

Deploy CSI Plugin on Kubernetes

Source: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md

KubeController Manager

API Server

etcd

MasterNode

CSI Volume Driver Container

external-snapshotterexternal-provisionerexternal-attacher(1 or more sidecar containers)

EmptyDir Volume

gRPC (UDS)

StatefulSet (replica: 1)

CSI Volume Driver Container

driver-registrar sidecar container

Host /var/lib/kubelet

UDS

DaemonSet PodKubelet

/…/sockets

UDS

UDS

Mountpropagation

Why Data Mobility

Why do we need Data Mobility

• Data mobility allows data to be accessible any time, any where, on premise or in the cloud.

• Use cases for data mobility by OpenSDSmulti-cloud data control– High availability– Data analytics– Data lifecycle management– ……

Use Case: High Availability

• Make data available at different geographic location.

• Make data available and accessible all the time without downtime.

Use Case: Data Analytics

• Upload datasets to pubic cloud for data analytics.

• Utilize compute resources in public cloud to do CPU intensive work.

0

1

2

3

4

5

6

Category1

Category2

Category3

Category4

Series 1 Series 2 Series 3

Use Case: Data Lifecycle Management• Frequently accessed data

stored on-premise.• Data not accessed for a long

time can be stored in archival storage in public cloud.

• Data can be moved to different location depending on usage, age, access frequency.

• Data can be deleted automatically based on policy.

Block StorageOn-Premise

Object Storage

Migration Policy

SnapshotPolicy:

Topology

Archive Policy

RetentionPolicy

On-Premise -> Public Cloud Archive DeleteUpload Snapshot

Policy-basedData Mobilityusing OpenSDS

OpenSDS Projects – Sushi and Hotpot

SUSHI The Northbound Plug-ins ProjectCommon plug-ins to enable OpenSDS storage services for cloud and application frameworks

HOTPOTThe Storage Controller Project

Single control for block, file, and object services across storage on premise and in clouds

OpenSDS HotpotDiscover, Pool, Provision, Orchestrate

D O C K – S T O R A G E D R I V E R S

SwordfishManilaNative Cinder Multi-Cloud

C O N T R O L L E R - S E R V I C E A P I

Block File Object

Direct AttachedStorage

Software-Defined Storage

EnterpriseStorage

CloudStorage

CLOUD ENTERPRISE

Plug-inPlug-inPlug-inPlug-inPlug-inPlug-in Plug-in Plug-in

OpenSDS Project - Gelato

App

X

Local CloudStorage(Object)

Replicator XX

X

X

X

X

Gelato The Multi-Cloud ProjectPolicy based multi-cloud data control to enable data mobility across clouds

Profiles and StorageClass/VolumeSnapshotClass

Enterprise Storage

Create PVC &VolumeSnapshot

K8S calls OpenSDSCSI Plugin

OpenSDS CSI Plugin callsOpenSDS controller to provisionvolume & snapshot

OpenSDS chooses storagebackend to create volume (PV) and

snapshot (VolumeSnapshotContent)

CommodityStorage

Cloud Storage

User

AdminCSI PluginOpenSDSCreate profiles Create StorageClass &

VolumeSnapshotClass

Admin

CSI Plugin and Policy-based Data Mobility• Create snapshot using OpenSDS CSI

plugin• Upload snapshot based on snapshot

topology• Migrate snapshot based on migration

policyOpenSDS Controller

Dock

Driver

Block Storage

Snapshot Profile

Object Storage

CSI PluginOpenSDS

Migration Policy

Snapshot Class

Migration Policy

Snapshot Topology

Volume Snapshot Example

Upload SnapshotCut Snapshot

Create Snapshot

apiVersion: snapshot.storage.k8s.io/v1alpha1kind: VolumeSnapshotmetadata:

name: new-snapshot-demonamespace: demo-namespace

spec:snapshotClassName: csi-opensds-snapclasssource:

name: opensdspvckind: PersistentVolumeClaim

apiVersion: snapshot.storage.k8s.io/v1alpha1kind: VolumeSnapshotClassmetadata:

name: csi-opensds-snapclasssnapshotter: csi-opensdspluginparameters:

profile: upload-snapshot-to-cloud

Note: Profile can be uuid or name of the profile in OpenSDS.

Multi-Cloud Data Control Architecture

Multi-Cloud Data Control Architecture

S3 Service Backend Service Dataflow Service

Database

Backend Adapters

GUI API Gateway

Data Mover

Identity Service

Multi-Cloud Backend Service

Backend Manager

API Handler

Backend

DB

Keystone

create/update/delete/get backend

auth

update metadata

check bucket

Multi-Cloud S3 Service

S3 Controller

API Handler

Backend

Location Mapper

DB

Backend Adapter

Keystone

get locationsearch

create/update/delete/get object/bucket

auth

update metadata

Multi-Cloud Dataflow Service

Connector Manager

API Handler

Migration Data Mover

Policy Manager

Job Manager

Plan Manager

Other Data Mover …

DB

Scheduler

Apache Kafka

DataFlow Manager

Keystone

put jobs

pull jobs

run plan

create/delete/update/getpolicy

create/delete/update/getconnector

create/delete/update/getplan

get jobsrestart/cancel job

get plancreate jobsupdate plan

auth

update job

Multi-Cloud Dashboard

Multi-Cloud Dashboard – Register Backend

Multi-Cloud Dashboard - Migration

Demo

• Take snapshot using OpenSDS CSI plugin• Snapshot uploaded to AWS S3• Migrate data from AWS S3 to Huawei

Cloud object store

OpenSDS Roadmap

OpenSDS Community

Getting Involved

• Repos: https://github.com/opensds• Slack: https://opensds.slack.com• Mailing list: https://lists.opensds.io• Weekly meetings:

https://github.com/opensds/design-specs/blob/master/README.md#opensds-technical-meetings