![Page 1: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/1.jpg)
PHP at 5000 Requests / SecHootsuite’s Scaling Story
Lead Technical Engineer - Platform@bmonkman
Bill Monkman
![Page 2: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/2.jpg)
![Page 3: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/3.jpg)
Overview - Selected Current Architecture
Users lb1 lb2 lb3 ... Nginx Load balancers
web1 web2 web3 ... Nginx web serversPHP-FPM PHP-FPM PHP-FPM PHP-FPM
mem1 ...
Memcached cluster
master slave
Mysql cluster
master slave
MongoDB cluster
slavemaster
shard1
shard2
geard1 geard2
Gearman cluster
worker1 ... ...
Services
...
![Page 4: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/4.jpg)
• Apache
• PHP
• MySQL
Technologies - at first
![Page 5: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/5.jpg)
Then...
![Page 6: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/6.jpg)
It’s hard to scale MySQL horizontally
Problem
![Page 7: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/7.jpg)
Memcached.
● Distributed cache, cluster of boxes with lots of RAM, trivial to scale
● Cache as much as possible, invalidate only when necessary
● Use cache instead of DB
● No joins - decouple entities (collection caching)
● Twemproxy!
Solution - Caching
![Page 8: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/8.jpg)
• Phil Karlton
“There are only two hard things in Computer Science: cache invalidation and naming things.”
![Page 9: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/9.jpg)
Solution - Caching
SocialNetwork
MvcModelMysql
MvcModelBase
MvcModelBaseCaching
![Page 10: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/10.jpg)
SELECT * FROM member WHERE org_id=888
set individual cache records
set collection cache
Automatic invalidation of collection cache
Solution - Caching
member_1 {data}
member_5 {data}
member_9 {data}
member_org_888 [1,5,9]
![Page 11: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/11.jpg)
It’s hard to scale MySQL horizontally
Now:
● No need to scale MySQL
● Able to serve the whole site on 1 MySQL server
● 500 MySQL SELECTs per second. 50,000 Memcached GETs.
● 99+% hit rate
Solution - Caching
![Page 12: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/12.jpg)
Then...
![Page 13: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/13.jpg)
Need a way to perform asynchronous, distributed tasks using a single-threaded language.
Problem
![Page 14: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/14.jpg)
Gearman.
● Distribute work to other servers to handle (workers also using
PHP, same codebase)
● Precursor to SOA where everything is truly distributed
● Many other solutions, queueing systems.
Solution - Gearman
![Page 15: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/15.jpg)
Solution - Gearman
geard1 geard2
gearworker1 gearworker2 gearworker6
![Page 16: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/16.jpg)
Need a way to perform asynchronous, distributed tasks using a single-threaded language.
Now:● Moved key tasks to Gearman
● Another cluster, scalable separately from web
● Discrete tasks, callable sync or async
Solution - Gearman
![Page 17: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/17.jpg)
Then...
![Page 18: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/18.jpg)
Need to store data with the potential to grow too big to handle effectively with MySQL.
Problem
![Page 19: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/19.jpg)
MongoDB.
● Certain data did not need to be highly relational
● NoSQL DB, many other solutions these days
● Mongo can be a pain, lots of moving parts
● Had to make our own sequencer where auto-incremented ids were
necessary
Solution - MongoDB
![Page 20: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/20.jpg)
Need to store data with the potential to grow too big to handle effectively with MySQL.
Now:● Multiple clusters containing amounts of data that likely would
have crushed MySQL● Billions of rows per collection, many TB of data on disk
Solution - MongoDB
![Page 21: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/21.jpg)
• Apache
• PHP
• MySQL
• Memcached
• Gearman
• MongoDB
Technologies
![Page 22: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/22.jpg)
Then...
![Page 23: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/23.jpg)
With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system?(SVN, big branches, merge hell)
Problem
![Page 24: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/24.jpg)
Dark Launching.
● Wrap code in block with a specific name
● That name will appear in a management page
● Can control whether or not that block is executed by modifying it’s value
● Boolean , random percentage, session-based, member list, organization
list, etc.
Solution - Dark Launching
![Page 25: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/25.jpg)
if (In_Feature::isEnabled(‘TWITTER_ADS’)) {
// execute new code
} else {
// execute old code
}
Solution - Dark Launching
![Page 26: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/26.jpg)
• Control your code
• Limit risk -> raise confidence -> speed up pace of releases
• “Branching in Production”
• Learning happens in Production
Dark Launching - Reasons
![Page 27: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/27.jpg)
With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system?
Now:● Work fast with more confidence
● Huge amount of control over production systems
● Typically 10+ code releases to production per day
● Push-based distribution with Consul
Solution - Dark Launching
![Page 28: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/28.jpg)
Then...
![Page 29: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/29.jpg)
With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code?
Problem
![Page 30: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/30.jpg)
Statsd / Graphite.Logstash / Elasticsearch / Kibana. Sensu
● Statsd for metrics
● Logstash for log events
● Sensu for monitoring / alerting
Solution - Monitoring
![Page 31: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/31.jpg)
Statsd::timing('apiCall.facebookGraph', microtime(true) - $startTime);
Solution - Monitoring
![Page 32: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/32.jpg)
Logger::event('user liked from in-stream', In_Log::CATEGORY_UX, $logData);
Solution - Monitoring
![Page 33: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/33.jpg)
• Visibility into the performance and behaviour of your application
• Iterate upon your code, measure results
• Pairs well with dark launching
• Also systems like New Relic
Solution - Monitoring
![Page 34: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/34.jpg)
With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code?
Now:● Able to watch performance / behaviour in real time.● Able to view important events both in the aggregate or very
granular● Able to control the system and watch the effect of changes
Solution - Monitoring
![Page 35: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/35.jpg)
Optimizations
![Page 36: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/36.jpg)
• Things expand beyond their initial scope
• Case in point: Translations
Optimizations
![Page 37: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/37.jpg)
![Page 38: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/38.jpg)
![Page 39: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/39.jpg)
• Within reason, push work up to users
• Make your users into a distributed processing grid
• e.g. Stream rendering
Optimizations - Push work to users
![Page 40: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/40.jpg)
• Performance is more important than clean code, business reqts (in the instances where they may be mutually exclusive)
• Fine line between future proofing and premature optimization
• Don’t add burdensome processes, but make it easy for your team to do things the right way
• Know your weak spots, protect against abuse
Optimizations - Performance / Risks
![Page 41: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/41.jpg)
![Page 42: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/42.jpg)
Technologies
LinuxNginx
MySQL
PHP-FPM
Jenkins
Scala
MongoDB
GearmanRedis
Akka
Python
Memcached
HAProxy
jQuery
ZeroMQ
RabbitMQBackbone
EC2Zend
Docker
Cloudfront CDN
VarnishElasticSearch
Logstash
Zookeeper
Kibana
Statsd/Graphite
Packer
Vagrant
Nagios
VirtualBox
Sensu
SymfonyRiak
Websockets
Comet
Hadoop
Spark/SharkAnsible
Consul
Composer
Git
RedshiftWebpack
![Page 43: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/43.jpg)
With a huge and growing monolithic codebase and over 80 engineers, how to keep scaling in a manageable way?
Problem
![Page 44: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/44.jpg)
SOA.
● Split up the system into independent services which communicate only via APIs
● Teams can work on their own services with encapsulated business logic and have their own
deployment schedules.
● We chose to use Scala/Akka for services, communicating via ZeroMQ
● SOA transition made easier by the “no joins” philosophy
● Tons of work
Solution - SOA
![Page 45: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/45.jpg)
SOM.
● “Service Oriented Monolith”
● When splitting up a monolithic codebase, dependencies are what kill you
● Fulfill dependencies by writing interim services using existing PHP code
● Maintain the contract and future scala services will be drop-in replacements
Solution - SOA
![Page 46: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/46.jpg)
With a huge and growing monolithic codebase and over 130 engineers, how to keep scaling in a manageable way?
Today:● Transitioning to Scala SOA● PHP will still be used as the Façade, a thin layer built on top of
the business logic of the services it interacts with.
Solution - SOA
![Page 47: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/47.jpg)
Conclusion
![Page 48: PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story](https://reader034.vdocuments.net/reader034/viewer/2022042518/55a20b641a28abbd4e8b45f3/html5/thumbnails/48.jpg)
Thank You!Bill Monkman@bmonkman
More Info:code.hootsuite.com