hopping in clouds - phpuk 17
TRANSCRIPT
Hopping
a tale of migration from one cloud provider to another
in Clouds
Michele OrselliCTO@Ideato
_orso_
micheleorselli / ideatosrl
Let’s start from the beginning…
What do we Italians like?
“Italians lose wars as if they were football matches, and football matches as if they were wars”
Winston Churchill
Peaks on gen - jun - aug up to 70 M pg/mth
Peaks during big matches
PaaS
IaaS
SaaS
PaaS
IaaS
SaaS
PaaS Platform as a Service
Zero configuration (almost)
Push the code “on the cloud” and you’re done
Hard limits on resource (e.g 50 db con)
Deploy via ftp on shared nfs dir (sf cache mess)
Blackbox: No realtime log, no access
PHP 5.3
Web: all articles categories, team details, ...
symfony1
Mobile: all articles categories, team details, ...
Sf2 components
Community Site with user generated content
Symfony2
μServices
Macro services
Talk: api for comments, votes, ratings
Symfony2
Adv: api for serving ads
Symfony2
Media: api for asset mgmt
Symfony2
The problems started with talk…
Quick wins
Tuning HTTP response headers
Caching more endpoints
Optimize queries
Tuning HTTP Headers
1 $date = new \DateTime(); 2 $date->modify("+$lifetime seconds"); 3 4 $response->setExpires($date); 5 $response->setMaxAge($lifetime); 6 $response->setSharedMaxAge($lifetime);
PaaS
IaaS
SaaS
PaaS
IaaS
SaaS
First candidate for migration was… the talk app
PHP from 5.3 to 5.6
Mysql from 5.0 to 5.6
Apache to nginx (+ php fpm)
Web servers ip are dynamic
Can connect only through bastion
Share user sessions between servers
Web servers ip are dynamic
Can connect only through bastion
Share user sessions between servers
1 'hosts' => function () { 2 $c = Ec2Client::factory([ 3 'profile' => 'calciomercato', 4 'region' => 'eu-central-1', 5 ]); 6 7 $ips = new GetInstancesIps($c); 8 9 return $ips->execute(); 10 }
1 public function execute() 2 { 3 $instances = $this->ec2Client 4 ->describeInstances( 5 [ 6 'DryRun' => false, 7 'Filters' => [ 8 [ 9 'Name' => 'instance.group-name', 10 'Values' => ['Web Public Auto-assign SG'], 11 ], 12 ], 13 ]); 14 15 return $instancesDescription->getPath( 16 'Reservations/*/Instances/*/NetworkInterfaces/*/PrivateIpAddresses/*/PrivateIpAddress' 18 ); 19 }
Web servers ip are dynamic
Can connect only through bastion
Share user sessions between servers
1 Host cmbastion 2 HostName xx.xx.xx.xx 3 User ec2-user 4 Port 9760 5 StrictHostKeyChecking no 6 UserKnownHostsFile /dev/null 7 IdentityFile ~/.ssh/cm_bastion.pem 8 LogLevel quiet
10 Host 10.0.14.* 11 User centos 12 StrictHostKeyChecking no 13 UserKnownHostsFile /dev/null 14 IdentityFile ~/.ssh/cm_production.pem 15 ProxyCommand ssh -W %h:%p cmbastion 16 LogLevel quiet 17 18 Host 10.0.24.* 19 User centos 20 StrictHostKeyChecking no 21 UserKnownHostsFile /dev/null 22 IdentityFile ~/.ssh/cm_production.pem 23 ProxyCommand ssh -W %h:%p cmbastion 24 LogLevel quiet
Web servers ip are dynamic
Can connect only through bastion
Share user sessions between servers
Nginx static cache
1 fastcgi_cache_key “$scheme$request_method$host$request_uri";
2 fastcgi_cache_lock on; 3 fastcgi_cache_revalidate on; 4 fastcgi_cache_valid 3m;
1 if ($request_method ~ ^(POST|PUT|DELETE)$ ) { 2 set $no_cache 1; 3 } 4 5 if ($request_uri ~* "/api/queue") { 6 set $no_cache 1; 7 } 8 9 location ~ ^/(app|dev)\.php(/|$) { [..] 17 18 # Enable fastcgi_cache 19 add_header X-Cache $upstream_cache_status; 20 fastcgi_cache CALCIOMERCATO_TALK; 21 fastcgi_cache_bypass $no_cache; 22 fastcgi_no_cache $no_cache; 23 }
8 9 location ~ ^/(app|dev)\.php(/|$) { 10 fastcgi_split_path_info ^(.+\.php)(/.*)$; 11 include fastcgi_params; 12 fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; 13 fastcgi_param DOCUMENT_ROOT $realpath_root; 14 fastcgi_param HTTPS off; 15 fastcgi_index app.php; 16 fastcgi_intercept_errors on; 17 18 # Enable fastcgi_cache [..] 23 }
Load test using old logs
Create AMI Images
Deploy latest version of the code
Switch DNS
The platform now runs on two clouds: RCS and AWS
Backup
DB and Machine snapshotted every night
Copied to another region
snapshot_id=` aws --profile snapshotdr rds describe-db-snapshots --db-instance-identifier $instance_identifier --region eu-central-1 | tail -n 1 | awk -F '{print $5}’ `
aws rds copy-db-snapshot --source-db-snapshot-identifier arn:aws:rds:eu-central-1:$id:snapshot:$snapshot_id --region eu-west-1 --target-db-snapshot-identifier mysqlrds-snap-copy-$NOW --copy-tags
snapshot_id=`aws --profile snapshotdr rds describe-db-snapshots --region eu-west-1 --db-instance-identifier $instance_identifier | head -n 1 | awk -F '{print $4}’`
aws rds delete-db-snapshot --region eu-west-1 --db-snapshot-identifier $snapshot_id
Adv
Only stateless apis
Small database small traffic
Infrastructure was already set
Easy!
Created a Cloudfront distribution for
dynamic content (adv.calciomercato.com)
Mobile
No database, it consumes data from other services
High impact, 40% of the total traffic
How to deal with static assets?
s3://com-calciomercato-cdn-mobile/
Created 2 Cloudfront distribution
dynamic content (m.calciomercato.com)
static content (cdnmobile.calciomercato.com)
Sync asset to s3 via s3cmd
s3cmd -m text/javascript --no-preserve sync /var/www/mobile/content/js s3://com-calciomercato-cdn-mobile/
Deploy on a sample machine
Performance test based on log
Deploy
Rebuilding AMI
Switch DNS
Tweaking assets caching
s3cmd --recursive modify --add-header='Cache-Control:max-age=3600' s3://com-calciomercato-cdn-mobile/assets/
Tweaking assets caching
s3cmd --recursive modify --add-header='Cache-Control:max-age=3600' s3://com-calciomercato-cdn-mobile/assets/
Less CF -> S3 requests Less $$$
Community
Allows uses to create their own personal blog
First app that can be considered “complete”
Exposes api for user related stuff
Works as SSO
Web servers ip are dynamic
Can connect only through bastion
Share user sessions between servers
1 services: 2 memcache: 3 class: Memcache 4 calls: 5 - [ addServer, [%memc_host%, %memc_port% ]] 6 session.handler.memcache: 7 class: 8 Symfony\Component\HttpFoundation\Session\Storage\Handler\MemcacheSessionHandler 10 arguments: [ 11 @memcache, 12 { prefix: %session_memcache_prefix%, 13 expiretime: %session_memcache_expire% } 14 ]
1 framework: 2 session: 3 handler_id: %session_handler_id%
Deal with User Generated Content
Created 2 Cloudfront distributions
dynamic content (vxl.calciomercato.com)
static content (cdnvxl.calciomercato.com)
Gaufrette
Filesystem abstraction layer
http://knplabs.github.io/Gaufrette/
1 knp_gaufrette: 2 adapters: 3 photo_storage: 4 aws_s3: 5 service_id: cdn.amazon_s3 6 bucket_name: %amazon_s3_bucket_name% 7 options: 8 directory: data 9 filesystems: 10 photo_storage: 11 adapter: photo_storage 12 alias: photo_storage_filesystem 13
8 cdn.amazon_s3: 9 class: Aws\S3\S3Client 10 factory_class: Aws\S3\S3Client 11 factory_method: 'factory' 12 arguments: 13 - 14 key: %cdn.amazon_s3.aws_key% 15 secret: %cdn.amazon_s3.aws_secret_key% 16 region: eu-central-1
7 class PhotoUploader 8 { [..] 27 public function upload(File $file, $dir) 28 { 29 $fullPath = $dir.'/'.$file->getFilename(); 30 31 if (!in_array($file->getMimeType(), self::$allowedTypes)) { 32 throw new \InvalidArgumentException($file->getMimeType()); 35 } 36 40 return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 ); 44 }
7 class PhotoUploader 8 { [..] 27 public function upload(File $file, $dir) 28 { 29 $fullPath = $dir.'/'.$file->getFilename(); 30 31 if (!in_array($file->getMimeType(), self::$allowedTypes)) { 32 throw new \InvalidArgumentException($file->getMimeType()); 35 } 36 40 return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 ); 44 }
return $this->filesystem->write( 41 $fullPath, 42 file_get_contents($file->getPathname()) 43 );
Deploy on a sample machine
Performance test based on log
Deploy
Rebuilding AMI
Copy User Assets on S3
Switch DNS
Web
Oldest and biggest codebase
Proxy for mobile calls
High traffic 60%
PHP 5.6 not supported by symfony 1
Plan A: try to upgrade sf1 to support php 5.6
Plan B: deploy web on different machines
1 protected function camelize($text) 2 { 3 return preg_replace(array('#/(.?)#e', '/(^|_|-)+(.)/e'), array("'::'. 4 strtoupper('\\1')", "strtoupper('\\2')"), $text); 5 } 6 7 public static function camelize($text) 8 { 9 return strtr(ucwords(strtr($text, array('/' => 10 ':: ', '_' => ' ', '-' => ' '))), array(' ' => '')); 11 }
Created 1 Cloudfront distribution for static content (cdnweb.calciomercato.com)
https://blog.cloudflare.com/zone-apex-naked-domain-root-domain-cname-supp/
calciomercato.com => cmelb-463612445.eu-central-1.elb.amazonaws.com
Deploy on a sample machine
Performance test based on log
Deploy
Rebuilding AMI
Switch DNS
Media
Only stateful api
Handles image thumbailing
Pretty big archive (70GB)
1 public function generateThumbAndUploadToCdn(File $file, $width, $height) 2 { 3 $downloadedFile = $this->downloadFromFileManager($file); 4 5 $cdnKey = $this->generateThumbCdnKey($file, $width, $height); 6 $resizedFile = $this->resizeFilesystemImage($downloadedFile, $width, $height); 8 9 $optimizedFile = $this->optimizeImage($resizedFile); 10 11 $this->uploadFileToCdn($optimizedFile, $cdnKey) 12 13 $this->updateFileInfoTumbs($file, $width, $height, $cdnKey); 14 15 $this->deleteTemporaryFile($downloadedFile); 16 $this->deleteTemporaryFile($optimizedFile); 17 18 return true; 19 }
Transfer from Rackspace CDN to S3
#!/bin/bash login="USERNAME_FTP" pass="FTP_PASSWORD" host="HOST_FTP_RACKSPACE" remote_dir='/web/content/data' local_dir=“/var/www/vhosts/media.calciomercato.pro/data"base_name="$(basename "$0")"
lftp -u $login,$pass $host << EOF set ftp:ssl-allow no set mirror:use-pget-n 5
mirror -c -P5 --log="/var/log/$base_name.log" "$remote_dir" "$local_dir"
quit
1 public function slugifyFilename($text) 2 { 3 $text = preg_replace('~[^\\pL\d]+~u', '.', $text); 4 $text = trim($text, '-'); 5 6 if (function_exists('iconv')) { 7 $text = iconv('utf-8', 'us - ascii//TRANSLIT', $text); 8 } 9 10 $text = preg_replace('~[^-\w\.]+~', '', $text); 11 12 return $text; 13 }
Monitoring
cloudwatch -> nagios -> catcicw retention max 2 sett
2 weeks retention: integration with cacti and nagios
...and Slack
Autoscaling
2 autoscaling groups
3 metrics
- CPU % (> 70%)
- Response Time (> 100 ms)
- # of req/sec (> 10000)
Full migration took 1 year
50% cost reduction
This talk is not about blaming RCS…
…simply it wasn’t suitable anymore for our needs :-)
Macro services FTW!
HTTP cache helped us a lot!
Measure measure measure
Rationalising api: less call, less $$$
Reorganise frontend stuff
Get rid of sf1
Upgrading to php7
Upgrading to ALB (HTTP/2)
Verona
JsDay: May 10th-11thPHPDay: May 12th-13th
Verona
Thank you!
https://joind.in/talk/cd6af
Michele OrselliCTO@Ideato
_orso_
micheleorselli / ideatosrl