hopping in clouds: a tale of migration from one cloud provider to another
Post on 12-Apr-2017
271 Views
Preview:
TRANSCRIPT
Hopping
a tale of migration from one cloud provider to another
inClouds
Let’s start from the beginning…
What is the national sport in Italy?
“Italians lose wars as if they were football matches, and football matches as if they were wars”
Winston Churchill
Peaks on gen - jun - aug up to 70 M pg/mth
Peaks during big matches
PaaS Platform as a Service
(almost) Zero configuration
Put the code “on the cloud” and you’re done
Hard limits on resource (e.g 50 db con)
Deploy via ftp (sf cache mess)
Blackbox: No realtime log, no access
PHP 5.3
Macro services
Web: the main web (sf1)
Mobile: mobile version (sf components)
Vxl: community site (sf2 v2.3)
Talk: api for comments, votes, ratings (sf2 v2.3)
Adv: api for ads serving (sf2 v2.3)
Media: api for images mgnt (sf2 v2.3)
The problems began with talk…
Quick wins
Tuning the http response headers
Caching more endpoints
Optimize queries
Tuning the HTTP Response
1 $date = new \DateTime(); 2 $date->modify("+$lifetime seconds"); 3 4 $response->setExpires($date); 5 $response->setMaxAge($lifetime); 6 $response->setSharedMaxAge($lifetime);
the first candidate for migration was… the talk app
PHP from 5.3 to 5.6
Mysql from 5.0 to 5.6
Apache to nginx (+ php fpm)
Web servers ip are dynamic
Can connect only through bastion
Share session between servers
Web servers ip are dynamic
Can connect only through bastion
Share user sessions between servers
1 'hosts' => function () { 2 $c = Ec2Client::factory([ 3 'profile' => 'calciomercato', 4 'region' => 'eu-central-1', 5 ]); 6 7 $ips = new GetInstancesIps($c); 8 9 return $ips->execute(); 10 }
1 public function execute() 2 { 3 $instances = $this->ec2Client 4 ->describeInstances( 5 [ 6 'DryRun' => false, 7 'Filters' => [ 8 [ 9 'Name' => 'instance.group-name', 10 'Values' => ['Web Public Auto-assign SG'], 11 ], 12 ], 13 ]); 14 15 return $instancesDescription->getPath( 16 'Reservations/*/Instances/*/NetworkInterfaces/*/PrivateIpAddresses/*/PrivateIpAddress' 18 ); 19 }
Web servers ip are dynamic
Can connect only through bastion
Share user sessions between servers
1 Host cmbastion 2 HostName xx.xx.xx.xx 3 User ec2-user 4 Port 9760 5 StrictHostKeyChecking no 6 UserKnownHostsFile /dev/null 7 IdentityFile ~/.ssh/cm_bastion.pem 8 LogLevel quiet
10 Host 10.0.14.* 11 User centos 12 StrictHostKeyChecking no 13 UserKnownHostsFile /dev/null 14 IdentityFile ~/.ssh/cm_production.pem 15 ProxyCommand ssh -W %h:%p cmbastion 16 LogLevel quiet 17 18 Host 10.0.24.* 19 User centos 20 StrictHostKeyChecking no 21 UserKnownHostsFile /dev/null 22 IdentityFile ~/.ssh/cm_production.pem 23 ProxyCommand ssh -W %h:%p cmbastion 24 LogLevel quiet
Web servers ip are dynamic
Can connect only through bastion
Share users sessions between servers
Nginx static cache
1 fastcgi_cache_key "$scheme$request_method$host$request_uri"; 2 fastcgi_cache_lock on; 3 fastcgi_cache_revalidate on; 4 fastcgi_cache_valid 3m;
1 if ($request_method ~ ^(POST|PUT|DELETE)$ ) { 2 set $no_cache 1; 3 } 4 5 if ($request_uri ~* "/api/queue") { 6 set $no_cache 1; 7 } 8 9 location ~ ^/(app|dev)\.php(/|$) { [..] 17 18 # Enable fastcgi_cache 19 add_header X-Cache $upstream_cache_status; 20 fastcgi_cache CALCIOMERCATO_TALK; 21 fastcgi_cache_bypass $no_cache; 22 fastcgi_no_cache $no_cache; 23 }
8 9 location ~ ^/(app|dev)\.php(/|$) { 10 fastcgi_split_path_info ^(.+\.php)(/.*)$; 11 include fastcgi_params; 12 fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; 13 fastcgi_param DOCUMENT_ROOT $realpath_root; 14 fastcgi_param HTTPS off; 15 fastcgi_index app.php; 16 fastcgi_intercept_errors on; 17 18 # Enable fastcgi_cache [..] 23 }
Load test using old logs
Create AMI Images
Deploy latest version of the code
Switch dns
Now we have the platform running on two clouds: RCS and AWS
Adv
Only stateless apis
Small database small traffic
Infrastructure was already set
Easy peasy
Created 1 Cloudfront distribution
dynamic content (adv.calciomercato.com)
Mobile
No database, it consumes data from other services
High impact, 40% of the total traffic
How to deal with static assets?
s3://com-calciomercato-cdn-mobile/
Created 2 Cloudfront distribution
dynamic content (m.calciomercato.com)
static content (cdnmobile.calciomercato.com)
Sync asset to s3 via s3cmd
s3cmd -m text/javascript --no-preserve sync /var/www/mobile/content/js s3://com-calciomercato-cdn-mobile/
Deploy on a sample machine
Performance test based on log
Deploy
Rebuilding AMI
Switch DNS
Community
Allows uses to create their own personal blog
First app that can be considered “complete”
Exposes api for user related stuff
Works as SSO
Web servers ip are dynamic
Can connect only through bastion
Share users sessions between servers
1 services: 2 memcache: 3 class: Memcache 4 calls: 5 - [ addServer, [%memc_host%, %memc_port% ]] 6 session.handler.memcache: 7 class: 8 Symfony\Component\HttpFoundation\Session\Storage\Handler\MemcacheSessionHandler 10 arguments: [ 11 @memcache, 12 { prefix: %session_memcache_prefix%, 13 expiretime: %session_memcache_expire% } 14 ]
1 framework: 2 session: 3 handler_id: %session_handler_id%
Deal with User Generated Content
Created 2 Cloudfront distributions
dynamic content (vxl.calciomercato.com)
static content (cdnvxl.calciomercato.com)
Gaufrette
Filesystem abstraction layer
http://knplabs.github.io/Gaufrette/
1 knp_gaufrette: 2 adapters: 3 photo_storage: 4 aws_s3: 5 service_id: cdn.amazon_s3 6 bucket_name: %amazon_s3_bucket_name% 7 options: 8 directory: data 9 filesystems: 10 photo_storage: 11 adapter: photo_storage 12 alias: photo_storage_filesystem 13
8 cdn.amazon_s3: 9 class: Aws\S3\S3Client 10 factory_class: Aws\S3\S3Client 11 factory_method: 'factory' 12 arguments: 13 - 14 key: %cdn.amazon_s3.aws_key% 15 secret: %cdn.amazon_s3.aws_secret_key% 16 region: eu-central-1
7 class PhotoUploader 8 { [..] 27 public function upload(File $file, $dir) 28 { 29 $fullPath = $dir.'/'.$file->getFilename(); 30 31 if (!in_array($file->getMimeType(), self::$allowedTypes)) { 32 throw new \InvalidArgumentException($file->getMimeType()); 35 } 36 40 return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 ); 44 }
7 class PhotoUploader 8 { [..] 27 public function upload(File $file, $dir) 28 { 29 $fullPath = $dir.'/'.$file->getFilename(); 30 31 if (!in_array($file->getMimeType(), self::$allowedTypes)) { 32 throw new \InvalidArgumentException($file->getMimeType()); 35 } 36 40 return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 ); 44 }
return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 );
Deploy on a sample machine
Performance test based on log
Deploy
Rebuilding AMI
Copy User Assets on S3
Switch DNS
Web
Oldest and biggest codebase
Proxy for mobile calls
High traffic 60%
PHP 5.6 not supported by symfony 1
Plan A: try to upgrade sf1 to support php 5.6
Plan B: deploy web on different machines
1 protected function camelize($text) 2 { 3 return preg_replace(array('#/(.?)#e', '/(^|_|-)+(.)/e'), array("'::'. 4 strtoupper('\\1')", "strtoupper('\\2')"), $text); 5 } 6 7 public static function camelize($text) 8 { 9 return strtr(ucwords(strtr($text, array('/' => 10 ':: ', '_' => ' ', '-' => ' '))), array(' ' => '')); 11 }
Created 1 Cloudfront distribution
static content (cdnweb.calciomercato.com)
https://blog.cloudflare.com/zone-apex-naked-domain-root-domain-cname-supp/
calciomercato.com => cmelb-463612445.eu-central-1.elb.amazonaws.com
Deploy on a sample machine
Performance test based on log
Deploy
Rebuilding AMI
Switch DNS
Media
Only stateful api
Handles image thumbailing
Pretty big archive (70GB)
1 public function generateThumbAndUploadToCdn(File $file, $width, $height) 2 { 3 $downloadedFile = $this->downloadFromFileManager($file); 4 5 $cdnKey = $this->generateThumbCdnKey($file, $width, $height); 6 $resizedFile = $this->resizeFilesystemImage($downloadedFile, $width, $height); 8 9 $optimizedFile = $this->optimizeImage($resizedFile); 10 11 $this->uploadFileToCdn($optimizedFile, $cdnKey) 12 13 $this->updateFileInfoTumbs($file, $width, $height, $cdnKey); 14 15 $this->deleteTemporaryFile($downloadedFile); 16 $this->deleteTemporaryFile($optimizedFile); 17 18 return true; 19 }
Transfer from Rackspace CDN to S3
#!/bin/bash login="USERNAME_FTP" pass="FTP_PASSWORD" host="HOST_FTP_RACKSPACE" remote_dir='/web/content/data' local_dir=“/var/www/vhosts/media.calciomercato.pro/data"base_name="$(basename "$0")"
lftp -u $login,$pass $host << EOF set ftp:ssl-allow no set mirror:use-pget-n 5
mirror -c -P5 --log="/var/log/$base_name.log" "$remote_dir" "$local_dir"
quit
1 public function slugifyFilename($text) 2 { 3 $text = preg_replace('~[^\\pL\d]+~u', '.', $text); 4 $text = trim($text, '-'); 5 6 if (function_exists('iconv')) { 7 $text = iconv('utf-8', 'us - ascii//TRANSLIT', $text); 8 } 9 10 $text = preg_replace('~[^-\w\.]+~', '', $text); 11 12 return $text; 13 }
Full migration took 1 year
from april 2015 to march 2016
50% cost reduction
This talk is not about blaming RCS…
…simply it wasn’t suitable anymore for our needs :-)
macro services FTW!
HTTP cache helped us a lot!
measure measure measure
Rationalizing api: less call, less $$$
Reorganize frontend stuff
Get rid of sf1
Upgrading to php7
Michele OrselliCTO@Ideato
_orso_
micheleorselli / ideatosrl
mo@ideato.it
Thank you!
https://joind.in/talk/1cf3c
Thanks to @dennais, @paolo, @alemazz, @ricfrank
top related