SYSTEM TESTING CLOUDSERVICES USING

EOS + CTA DEVELOPMENT USE-CASE from IT STorage group Julien Leduc CERN

DATA ARCHIVING AT CERNAd aeternum storage7 tape libraries, 83 tape drives, 20k tapesCurrent use: 180 PBCurrent capacity: 0.6 EBExponentially growing

DATA ARCHIVING AT CERNEVOLUTION

EOS + tapes...EOS is CERN strategic storage platformtape is the strategic long term archive medium

EOS + tapes = ♥Meet CTA: CERN Tape ArchiveStreamline data paths, so�ware andinfrastructure

CTA is glued to the back of EOSEOS manages CTA tape files as replicasCTA contains a catalogue of all tape filesCTA provides optimised, preemptive scheduling

CTA DEVELOPMENT TIMELINEEnd 2016: First functional prototype releaseApril 2017: First release for additional copy use cases2018: Production-ready version

Easy migration path from CASTOR to EOS+CTA: onlymetadata need to be migrated CASTOR tape formatwill be reused.

CTA + EOS DEVELOPMENTSThis involves tightly coupled development in the intial

phase for both so�ware, and extensive testing toquickly catch regressions.

CASTOR INTEGRATION TESTSEasy situation:

all components are within one git repositoryPuppet deploys development instances on VMsLimited external dependencies per instance: 1database, 1 virtual tape library

CASTOR INTEGRATION TESTSBut several issues:

deploying a developer instance from scratch takesloooonnng time...code changes in CASTOR o�en require Puppetmanifest changereal tape hardware tests are way further downthe road in separate hostgroups, environments...

which implies ad hoc developer tests...

CTA+EOS INTEGRATION TESTSComplex situation:

2 distinct so�ware projectsMore external dependencies per instance: 1database, 1 virtual tape library, 1 objectstore

CTA+EOS INTEGRATION TESTSHow to fix everything?

I am lazy and impatientno manual operation → CImake it fast

Must allow similarly easy beta testingdeployments for administrators/users (simple andbulletproof)How to test real tape hardware?

CTA CIImplemented in CERN Gitlab instance

Build so�ware: CTA RPMs available as artifactsBuild and publish a generic Docker image in gitlabregistry

Contains all required RPMs for instantiation (CTAartifacts, specific EOS version, specific XROOTDversion)

Run system tests in custom kubernetes cluster

BASIC KUBERNETES CONCEPTSNAMESPACE 1: subnet 10.0.1.x

CONFIGURATIONs

Config1

SERVICE 1

SERVICE 2

SERVICEs

POD 1

POD2

POD3

PODsPODs

NAMESPACE 2: subnet 10.0.2.x

SERVICE 1

SERVICE 2

SERVICEs

POD 1

POD2

PODsPODs

kubernetes CLUSTER

PV1

label: ceph

PV2

label: local

KUBERNETES RESOURCESSystem tests on dedicated kubernetes clusters

One Puppet deployed kubernetes cluster perdeveloper on one VMKubernetes resources per cluster:

1 Oracle database (+ unlimited sqlite accounts)1 Ceph objectstore (+ unlimited localobjectstores)10 Virtual tape libraries: 2 tape drives, 10 tapes

INSTANTIATING A TESTCreate k8 NamespaceInstantiate all Services in the namespaceConsumable resources are implemented asPersistent Volumes

Issue a Persistent Volume Claim with selectorInstantiate associated Configuration in theNamespace

Instantiate all the Pods with their associatedcontainers to implement all the servicesWait for all the pods to be ready

NAMESPACE

CONFIGURATIONs

ObjectStore

DataBase

TapeLibrary

ctafrontend

ctaeos

kdc

SERVICEs

SYSTEM TEST

setup EOS WFExrdcp file -> ctaeos- is it on tape?remove EOS disk copyretrieve from CTA- is is back in EOS?

GITLAB

CTA frontend

CTA EOS

CTA CLI

tape srv

KDC

PODsPODs

INSTANTIATING A TEST

REAL TAPE DRIVE TESTSDeploy Puppet manifest on real hardwareAdd physical tape library resources in hieraIncrease timeouts for system tests

VOILÀ!We can deploy the same kubernetes instance on real

tape hardware and run exactly the same system tests.

THE ENDVery powerful approach addresses all our use casesFast, flexible, isolated and self contained in so�warerepository

TO DOWrite more system tests (using jtest? scripts?...)Bulletproofing reproducibility for regression testsEvaluate possible production use ☺

QUESTIONS?