fowa scaling the lamp stack workshop

Scaling the LAMP StackFuture of Web AppsOctober 5, 2007

Introductions

Specific Problems, Challenges and Issues

About this workshop

• This is a broad topic

• Theory and application

• Real-world focus

• Interactive (please!)

About web apps and scaling

Some different ways of looking at the problem…

Things to think about• Multi-server: locking and

concurrency• Running many: keep in mind

what’s expensive, sloppy or risky• Code quality• The law of truly large numbers

Elements of Scaling• Split up different tasks• Use more hardware (intelligently)• Partition• Replicate• Cache• Optimize (code and hardware)• Identify and fix weaknesses• Manage

Tools and Components

• Apache + PHP

• MySQL

• File System (local)

• Networked File System

• Load Balancers

• memcached

Contemplating Scaling

• Understand what your app does (and how much)

• Identify the bottlenecks

• Solve near-term problems

• Design well, but don’tover-design

Web apps do lots of things

Different operations have different scaling issues.

What does your app do?

List the high level elements of what your application does. Separate out different functions that will have different scaling issues.

Common things that web apps do

• Manage connections/protocols

• Deliver static content

• Manage sessions

• Manage user data

• Render dynamic pages

• Access external APIs

• Process media

Update the list of things your app does

• Add anything you missed

• Note which items you do in quantity

Easy vs. Difficult Scaling

What happens when you add hardware?

•Does it work?

•Does more hardware = more performance?

Things that break when you scale

• State that isn’t properly shared (especially sessions)

• Updates/refreshes (caching and replication issues)

Things that don’t improve when you add more servers

• Unpartitioned databases

• Anything that locks/blocks

• Inefficient code, especially big queries

Scaling Each Element

• (do easy separations first)

Managing Connections/Protocols

• No problem putting on multiple servers

• Apache is goodo Not too far away out of the boxo Moderately tunable

• Linux tuningo TCP stack (tune to handle unusual

networking needs)

Key Apache Configuration Issues

• MaxClients (and ServerLimit, ThreadLimit and ThreadsPerChild)

• Avoid using PHP (or other) handler unnecessarily

• Use the worker MPM

• Maybe MaxRequestsPerChild

Delivering Static Content

• Don’t process it unnecessarilyo Either cache or use no Apache

handlerso Caching can let you treat semi-static

content as static

• Multiple servers complicates updates, but is otherwise easy

General Discussion:Multi-server, state and sessions

Rethinking state for multi-server environments

• What is state?

• Short-term state (sessions)

• Long-term state (application data)

• Managing state is usually the hardest part of scaling

What happens with state

• Written (created/destroyed/changed)

• Read

• Stored

Requirements for managing state

• Depend on what it is and how it is used

• Perfect coherence

• Performance of different operations

Ways of scaling state

• Replication: make more copies

• Partitioning: split up the work

• Caching

Should make different choices for different state/data elements

About Load Balancers

• What load balancers doo Spread loado Detect server failureso Stickiness/persistenceo Acceleration (especially SSL)

• Fancy features (including good stickiness) are expensive

Why sticky sessions are not usually good in practice

• Servers fail

• Corner cases exist

Managing Sessions

Where session data can be stored

• Browser cookies

• Web server temporary files(not scalable)

• App server state

• Database

• Cache

PHP session management

• Default (files) method is not multi-server friendly, and thus not scalable (unless sticky)

• Can implement a different back-end easily

Designing a session back-end

• Requirements

• Data storage optionso Cookies only (re-auth, let the browser

take care of the logout – but less secure)o Full-featured involves a combination of

cookies and database and cache

(discussion of session details)

Managing small user data

• Databases are more efficient, flexible and sharable than small files

• Frequently-read data should be cached

Managing large user data

• NFS has flaws but is almost inevitable

• Locking is usually not important, but can be

• Performance degradation can be sudden

About NFS• NFS is usually transparent to your

app• NFS is easy to implement gives you

multiple-write access• NFS locking is not to be trusted• The Linux NFS client is slow for writes

and can do bad things under stress

User data and locking

• Names based on hashes often mean no locking is needed

• Databases do locking better than file systems do

• Locking requires housekeeping

Disk Storage Hardware

• Disk performance can degrade suddenly

• If the ratio of access to storage is low, then even slow disk is usually fine

• Think about seek times and spindles

Rendering dynamic pages• Depends heavily on application

specifics (query, search, process, etc.)

• Watch out for:o Onerous queries (create and watch slow query log)o Locking of resources and/or incoherence if state changeso Heavy CPU and memory usage

• Cache both elements and complete pages

Processing media

• CPU intensive

• May be memory intensive

• Might be spiky

• Might need its own server pool

Hardware

• Start simple

• Observe performance and respond accordingly

• Get lots of memory

Hardware-driven behaviors

• Sudden degradation because demand exceeds supply (usually relieved unhappily)

• Get behind due to a spike, and recover

• Not enough resources for normal optimization

Specific hardware issues

• Not enough memoryo Severe: paging/swappingo Mild: poor automatic caching; slowness

due to fragmentation

• Disk seek (very common)

• CPU (but might really be memory)

• Disk throughput (rare for web apps)

Hardware decisions

• SCSI/SAS vs. SATA

• Resource ratios

• Combining vs. splitting functions

• Big vs. little boxes

Techniques

• Caching

• Partitioning

• Replication

• Data management middleware

• Queuing

Caching• Turn expensive operations in to

cheap ones• Reduce:

o Database readso Object and page calculation/rendering

operations

• Cache objects and subobjects• Add memory

Apache Caching

• Can be done with zero application modifications

• Complete pages/HTTP requests only

• Must use Apache 2.2

• Cache is not shared between servers

memcached

• Extremely useful

• Distributed caching system

• Requires new thinking and new coding

• Straightforward API

memcached URLs

• Home: http://www.danga.com/memcached/

• Intro: http://www.majordojo.com/2007/03/memcached-howto.php

• PHP documentation: http://www.php.net/memcache

Partitioning• Mostly for data management

• Split load on to separate servers/pools

• Partition algorithm/mechanism must be lightweight

• Partition algorithm must anticipate the future

File Storage Partitioning

• Index/database gives the most flexibility

• Hash-based is simplest

Database partitioning

• You will need to do this, but perhaps later than you think

• Index vs. hash-based

Replication

• Used where data is read far more than written

• Consider caching first

• Also used for failure recovery

Types of Replication

• Replication: sync vs. asynco Synchronous is not usually scalableo Asynchronous only works with certain

kinds of data and use cases, because of coherence issues

Database Replication

• Simple but finicky

• Asynchronous (but not by much)

• Allows big queries and backups to be moved to separate servers

File System Replication

• Slow and very asynchronous

• Mostly for disaster recovery

Data Management Middleware• Mostly for databases• Can handle partitioning and

replication, and do it well• Big investment in coding to the

API• Sometimes easier to add

functionality to app

Queuing

• Save work for later

• Useful for less urgent operations, especially messaging

• Can be used to wait for a pause, or to separate hardware

Dealing with lots of hardware(operations)

• Automation

• Process

Imaging/Provisioning

• Be consistent

• Use your distro’s automation (Kickstart, AutoYaST, etc.)

• Use boring, meaningful hostnames

• Make re-imaging easy

Deployment Systems• Content and code replication• Coherence/atomic updates• Managing pieces and processes• Simple scripts are fine• Create audit trail• Include back-out• Think 3AM• Do it!

Monitoring systems

• A pain, but a lifesaver• Start with built-in basics• Add custom checks, especially end-

to-end and communication between pieces

• Eliminate false alarms (ongoing)• Nagios, usually

Coping with hardware failure• Have extra servers/capacity• Load balancers handle stateless

layers• Replication prepares you to handle

data layers manually• Use middleware or app-level multiple

writes to get true data layer redundancy

Change management

• Part automation, part process

• Use version control on everything

• Stage changes with realistic data

• Know how to back out

• Consult the right people (internal and/or external)

Efficiency

• Access the smallest amount (DB, FS, etc.)

• Don’t do complex stuff when simple will suffice

Using the database efficiently• Keep it simple• Know what queries you do• Index every query key• Cache to reduce demand• Check slow query log• Replicate if you need big queries

The messy real world

Security and abuse

• Mostly same issues, just magnified

• You will be a target

• Spam (coming and going)

• Abuse of file storage

Corner Cases

• Murphy’s law enforcement

• Watch out for how different user activities relate

• Lock data, not functions

• Housekeeping

Performance and tuning• Observation and responsiveness is

more important than pre-optimizing• Redesign as needed• Collect the data to be able to

analyze (both resource utilization and end-user performance)

Miscellaneous Warnings

Files and directories

• Most default file system configurations get really slow with lots of files in one directory

• Numerical limits on files and subdirectories

• Some programs don’t like files over 2GB

AJAX

• Sequential round trips

• Make preloading invisible

• UI that waits for too many things

Other topics

• Multiple sites

• CDNs

Scaling the LAMP StackFuture of Web AppsOctober, 2007

Daniel Lieberman daniel@bitpusher.com