Hopping

a tale of migration from one cloud provider to another

inClouds

Michele OrselliCTO@Ideato

_orso_

micheleorselli / ideatosrl

mo@ideato.it

Let’s start from the beginning…

What is the national sport in Italy?

“Italians lose wars as if they were football matches, and football matches as if they were wars”

Winston Churchill

Peaks on gen - jun - aug up to 70 M pg/mth

Peaks during big matches

PaaS Platform as a Service

(almost) Zero configuration

Put the code “on the cloud” and you’re done

Hard limits on resource (e.g 50 db con)

Deploy via ftp (sf cache mess)

Blackbox: No realtime log, no access

PHP 5.3

Macro services

Web: the main web (sf1)

Mobile: mobile version (sf components)

Vxl: community site (sf2 v2.3)

Talk: api for comments, votes, ratings (sf2 v2.3)

Adv: api for ads serving (sf2 v2.3)

Media: api for images mgnt (sf2 v2.3)

The problems began with talk…

Quick wins

Tuning the http response headers

Caching more endpoints

Optimize queries

Tuning the HTTP Response

1 $date = new \DateTime(); 2 $date->modify("+$lifetime seconds"); 3 4 $response->setExpires($date); 5 $response->setMaxAge($lifetime); 6 $response->setSharedMaxAge($lifetime);

the first candidate for migration was… the talk app

PHP from 5.3 to 5.6

Mysql from 5.0 to 5.6

Apache to nginx (+ php fpm)

Web servers ip are dynamic

Can connect only through bastion

Share session between servers

Web servers ip are dynamic

Can connect only through bastion

Share user sessions between servers

1 'hosts' => function () { 2 $c = Ec2Client::factory([ 3 'profile' => 'calciomercato', 4 'region' => 'eu-central-1', 5 ]); 6 7 $ips = new GetInstancesIps($c); 8 9 return $ips->execute(); 10 }

1 public function execute() 2 { 3 $instances = $this->ec2Client 4 ->describeInstances( 5 [ 6 'DryRun' => false, 7 'Filters' => [ 8 [ 9 'Name' => 'instance.group-name', 10 'Values' => ['Web Public Auto-assign SG'], 11 ], 12 ], 13 ]); 14 15 return $instancesDescription->getPath( 16 'Reservations/*/Instances/*/NetworkInterfaces/*/PrivateIpAddresses/*/PrivateIpAddress' 18 ); 19 }

Web servers ip are dynamic

Can connect only through bastion

Share user sessions between servers

1 Host cmbastion 2 HostName xx.xx.xx.xx 3 User ec2-user 4 Port 9760 5 StrictHostKeyChecking no 6 UserKnownHostsFile /dev/null 7 IdentityFile ~/.ssh/cm_bastion.pem 8 LogLevel quiet

10 Host 10.0.14.* 11 User centos 12 StrictHostKeyChecking no 13 UserKnownHostsFile /dev/null 14 IdentityFile ~/.ssh/cm_production.pem 15 ProxyCommand ssh -W %h:%p cmbastion 16 LogLevel quiet 17 18 Host 10.0.24.* 19 User centos 20 StrictHostKeyChecking no 21 UserKnownHostsFile /dev/null 22 IdentityFile ~/.ssh/cm_production.pem 23 ProxyCommand ssh -W %h:%p cmbastion 24 LogLevel quiet

Web servers ip are dynamic

Can connect only through bastion

Share users sessions between servers

Nginx static cache

1 fastcgi_cache_key "$scheme$request_method$host$request_uri"; 2 fastcgi_cache_lock on; 3 fastcgi_cache_revalidate on; 4 fastcgi_cache_valid 3m;

1 if ($request_method ~ ^(POST|PUT|DELETE)$ ) { 2 set $no_cache 1; 3 } 4 5 if ($request_uri ~* "/api/queue") { 6 set $no_cache 1; 7 } 8 9 location ~ ^/(app|dev)\.php(/|$) { [..] 17 18 # Enable fastcgi_cache 19 add_header X-Cache $upstream_cache_status; 20 fastcgi_cache CALCIOMERCATO_TALK; 21 fastcgi_cache_bypass $no_cache; 22 fastcgi_no_cache $no_cache; 23 }

8 9 location ~ ^/(app|dev)\.php(/|$) { 10 fastcgi_split_path_info ^(.+\.php)(/.*)$; 11 include fastcgi_params; 12 fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; 13 fastcgi_param DOCUMENT_ROOT $realpath_root; 14 fastcgi_param HTTPS off; 15 fastcgi_index app.php; 16 fastcgi_intercept_errors on; 17 18 # Enable fastcgi_cache [..] 23 }

Load test using old logs

Create AMI Images

Deploy latest version of the code

Switch dns

Now we have the platform running on two clouds: RCS and AWS

Adv

Only stateless apis

Small database small traffic

Infrastructure was already set

Easy peasy

Created 1 Cloudfront distribution

dynamic content (adv.calciomercato.com)

Mobile

No database, it consumes data from other services

High impact, 40% of the total traffic

How to deal with static assets?

s3://com-calciomercato-cdn-mobile/

Created 2 Cloudfront distribution

dynamic content (m.calciomercato.com)

static content (cdnmobile.calciomercato.com)

Sync asset to s3 via s3cmd

s3cmd -m text/javascript --no-preserve sync /var/www/mobile/content/js s3://com-calciomercato-cdn-mobile/

Deploy on a sample machine

Performance test based on log

Deploy

Rebuilding AMI

Switch DNS

Community

Allows uses to create their own personal blog

First app that can be considered “complete”

Exposes api for user related stuff

Works as SSO

Web servers ip are dynamic

Can connect only through bastion

Share users sessions between servers

1 services: 2 memcache: 3 class: Memcache 4 calls: 5 - [ addServer, [%memc_host%, %memc_port% ]] 6 session.handler.memcache: 7 class: 8 Symfony\Component\HttpFoundation\Session\Storage\Handler\MemcacheSessionHandler 10 arguments: [ 11 @memcache, 12 { prefix: %session_memcache_prefix%, 13 expiretime: %session_memcache_expire% } 14 ]

1 framework: 2 session: 3 handler_id: %session_handler_id%

Deal with User Generated Content

Created 2 Cloudfront distributions

dynamic content (vxl.calciomercato.com)

static content (cdnvxl.calciomercato.com)

Gaufrette

Filesystem abstraction layer

http://knplabs.github.io/Gaufrette/

1 knp_gaufrette: 2 adapters: 3 photo_storage: 4 aws_s3: 5 service_id: cdn.amazon_s3 6 bucket_name: %amazon_s3_bucket_name% 7 options: 8 directory: data 9 filesystems: 10 photo_storage: 11 adapter: photo_storage 12 alias: photo_storage_filesystem 13

8 cdn.amazon_s3: 9 class: Aws\S3\S3Client 10 factory_class: Aws\S3\S3Client 11 factory_method: 'factory' 12 arguments: 13 - 14 key: %cdn.amazon_s3.aws_key% 15 secret: %cdn.amazon_s3.aws_secret_key% 16 region: eu-central-1

7 class PhotoUploader 8 { [..] 27 public function upload(File $file, $dir) 28 { 29 $fullPath = $dir.'/'.$file->getFilename(); 30 31 if (!in_array($file->getMimeType(), self::$allowedTypes)) { 32 throw new \InvalidArgumentException($file->getMimeType()); 35 } 36 40 return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 ); 44 }

7 class PhotoUploader 8 { [..] 27 public function upload(File $file, $dir) 28 { 29 $fullPath = $dir.'/'.$file->getFilename(); 30 31 if (!in_array($file->getMimeType(), self::$allowedTypes)) { 32 throw new \InvalidArgumentException($file->getMimeType()); 35 } 36 40 return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 ); 44 }

return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 );

Deploy on a sample machine

Performance test based on log

Deploy

Rebuilding AMI

Copy User Assets on S3

Switch DNS

Web

Oldest and biggest codebase

Proxy for mobile calls

High traffic 60%

PHP 5.6 not supported by symfony 1

Plan A: try to upgrade sf1 to support php 5.6

Plan B: deploy web on different machines

https://github.com/LExpress/symfony1

1 protected function camelize($text) 2 { 3 return preg_replace(array('#/(.?)#e', '/(^|_|-)+(.)/e'), array("'::'. 4 strtoupper('\\1')", "strtoupper('\\2')"), $text); 5 } 6 7 public static function camelize($text) 8 { 9 return strtr(ucwords(strtr($text, array('/' => 10 ':: ', '_' => ' ', '-' => ' '))), array(' ' => '')); 11 }

Created 1 Cloudfront distribution

static content (cdnweb.calciomercato.com)

https://blog.cloudflare.com/zone-apex-naked-domain-root-domain-cname-supp/

calciomercato.com => cmelb-463612445.eu-central-1.elb.amazonaws.com 

Deploy on a sample machine

Performance test based on log

Deploy

Rebuilding AMI

Switch DNS

Media

Only stateful api

Handles image thumbailing

Pretty big archive (70GB)

1 public function generateThumbAndUploadToCdn(File $file, $width, $height) 2 { 3 $downloadedFile = $this->downloadFromFileManager($file); 4 5 $cdnKey = $this->generateThumbCdnKey($file, $width, $height); 6 $resizedFile = $this->resizeFilesystemImage($downloadedFile, $width, $height); 8 9 $optimizedFile = $this->optimizeImage($resizedFile); 10 11 $this->uploadFileToCdn($optimizedFile, $cdnKey) 12 13 $this->updateFileInfoTumbs($file, $width, $height, $cdnKey); 14 15 $this->deleteTemporaryFile($downloadedFile); 16 $this->deleteTemporaryFile($optimizedFile); 17 18 return true; 19 }

Transfer from Rackspace CDN to S3

#!/bin/bash login="USERNAME_FTP" pass="FTP_PASSWORD" host="HOST_FTP_RACKSPACE" remote_dir='/web/content/data' local_dir=“/var/www/vhosts/media.calciomercato.pro/data"base_name="$(basename "$0")"

lftp -u $login,$pass $host << EOF set ftp:ssl-allow no set mirror:use-pget-n 5

mirror -c -P5 --log="/var/log/$base_name.log" "$remote_dir" "$local_dir"

quit

1 public function slugifyFilename($text) 2 { 3 $text = preg_replace('~[^\\pL\d]+~u', '.', $text); 4 $text = trim($text, '-'); 5 6 if (function_exists('iconv')) { 7 $text = iconv('utf-8', 'us - ascii//TRANSLIT', $text); 8 } 9 10 $text = preg_replace('~[^-\w\.]+~', '', $text); 11 12 return $text; 13 }

Full migration took 1 year

from april 2015 to march 2016

50% cost reduction

This talk is not about blaming RCS…

…simply it wasn’t suitable anymore for our needs :-)

macro services FTW!

HTTP cache helped us a lot!

measure measure measure

Rationalizing api: less call, less $$$

Reorganize frontend stuff

Get rid of sf1

Upgrading to php7

Michele OrselliCTO@Ideato

_orso_

micheleorselli / ideatosrl

mo@ideato.it

Thank you!

https://joind.in/talk/1cf3c

Thanks to @dennais, @paolo, @alemazz, @ricfrank