Building a Personal Cloud Storage ServiceVolkan EsgelTurkcellAugust 24, 2016

2

About Me

Volkan EsgelSenior Software Engineer

TURKCELL

3

About Turkcell

* http://investor.turkcell.com.tr/2015/turkcell-group

Integrated communication and technology services

player in TurkeyTurkcell Global

9 countries

68,9 million total subscribers1,5 million fixed

subscribers 600 thousand TV subscribers

IntroductionLegacy Solution Current Solution

5

Legacy Solution

Adding features was costly (time & budget) User Experience

not good

No Folder Structureonly tagging

Security Issues

6

Current Solution

Distributed

Fast

Fault Tolerant

Highly Scalable

Extendable

New features can be added easily

7

Technologies

Spring FrameworkOpenStack

Keystone & SWIFTElasticSearch

RabbitMQ

Oracle DB ImageMagick

FFmpeg

8

9

OpenStack Projects

10

Main Projects

Business OpenStack

11

* Keystone v2.0 (with OS-KSADM extension) / SWIFT v1

Account (Project / Tenant)

User

Container(s)

Main Extended

12

Containers

Main Container

• Main Storage• UUID as filename

Extended Container

• Thumbnail• Video Preview• Profile Photo

13

Uploading a File

Client

Oracle DB

Transcoding

ImageMagickFFmpeg

SWIFT

14

Temporary URL

Adding X-Auth-Token to the request header

not possible for all cases

Temporary URLfrom security perspective

Our Usage Cases

15

Our Usage

33 OpenStack Servers

3.3 PB Storage Space

6 M Daily File Upload

1.6 B Total Files

OpenStack MiddlewareCustom middleware modules

17

Keystone

Business & OpenStackmust be use the sameauthentication token

Several authentication methods

Turkcell Auth, Mobile Network Auth, Remember Me, etc.

Authentication methodsshould be

easily extensible

Solution for these casesnot easy

on Keystone Side

Custom Keystone Middleware

authenticates via

RESTful API

18

Keystone

No need to accessKeystone

from Internet

CallKeystone Auth APIfrom Business API

19

Authentication

API- BUSINESS -

Keystone- OPENSTACK -

Authenticate User- BUSINESS -

Token Cache- BUSINESS -

USER- CLIENT -

20

SWIFT

Client Sync Middleware

Notification MiddlewareSecurity Middleware

21

SWIFT – Notification Middleware

Notify BACKEND

about file uploads

No failure

any uploaded fileyet

Transfer notificationsover RabbitMQ

Python Kombu

Get custom paramsusing X-Object-Meta-*

headers

Only forMain Container

22

SWIFT – Security Middleware Open Internet

MAIN

Only GET requestsare allowed

EXTENDED

Define IP Blocks ofInternal Servers

in conf file

Reject invalidPUT requests

( X-Object-Meta-File-Name header required )

Allow onlyOBJECT operations

BlockACCOUNT & CONTAINER

operations

Only PUT, GET & OPTIONS requests are allowed

23

SWIFT – Client Sync Middleware

PUT

X-Meta-Strategy:0 Check for conflict

1 Override existing object

X-Meta-Recent-Server-Hash:Known ETag value of object on the server

X-Meta-Recent-Server-Hash & ETag :

equals no conflict, allow PUT request and update existing one

not equals conflict, return bad response with status

USER- CLIENT -

File SystemAdvantages of using custom filesystem on DB instead of SWIFT Object Paths & Container Listing

25

File System

All Objectslocated under root path of

the container

Object NamesUUIDs

Display NameMetadata Header

File ListingsOracle DB

Photo & Video ListingElasticSearch Unified (Metadata) Search

ElasticSearch

26

Difficulties of SWIFT File System

File Statisticsuser and/or content based file

statistics

New Featuresadding new features to

the filesystem

Pseudo Folder (Virtual)renaming a folder requires copying all sub-objects and

deleting old files – costlyDropbox & Google Drive

27

Conclusion

Developed a Personal Cloud Storage Servicejust in 6 months

from scratchNo critical security

issue is found

tested multiple times by the internal & independent security organizations

No vendor lock-inhardware / software

HighlyScalable & Extendable

DemonstrationFinal Product

29

30

31

32

33

?https://akillidepo.turkcell.com.tr

Turkcell Akıllı Depo