Large Scale Web Development

Theory and practice with Java

2/3/2011 Michaël Figuière

Scalability best practices

Typical Web Architecture

Backend A

Load BalancerApplication

Instance

ApplicationInstance

Backends may be slow, fast, highly available or not

Backend B

Backend C

Backend D

Backend E

ApplicationInstance

Facing the network’s reality

• Some requests will be slow

• Some requests won’t answer

• Some requests will just fail

Server / proxy overloaded, network traffic, ...

Server failure, network failure, OS and JVM pressure

Server application’s bugs, GC, connection rejected...

Handling the network’s reality

• Timeout must be set and handled for every remote request

• Use Circuit Breaker pattern

• Setting a deadline for your answer may be helpful

If API doesn’t offer it, ExecutorService and Future can help

Whatever happen, answer will be sent as is within N sec. If mandatory goals aren’t achieved, return error.

Avoid requesting an already overloaded service

Requests make the load

• Two requests instead of one double the load of the backend

• Cache must be sized with care

Here, counting requests isn’t about optimization, it’s critical

Cache Misses increase load on backends

Make requests in parallel

• Parallel requests reduce overall duration

• Thread pools make it possible easily

• Thread pools also act as a throttle to shield a backend

Mandatory when backends are slow

No more than N concurrent requests

ExecutorService and Future do the job

Make requests in parallel

D = Sum of requests durations

D = Max of requests durations

From separated thread pools to semaphores

• When you have a thousand threads, merging thread pools can help

• A semaphore can then do the throttling job

• Semaphore can also be tuned for live throttle tuning !

Mutualize resources

Allowing to slow down a requests stream to a dying backend

Limit the concurrent users of a resource

Serialized caches in Java heap

• Garbage Collector tuning can be time consuming

• Serializing data structures in Java heap caches reduces pressure on GC

• Don’t use Java Standard Serialization, use Avro, Kryo, or ProtocolBuffer

Especially when production environment is hard to simulate

Very low CPU overhead and a so compact format

GC time complexity partly depends on amount of references

Memcached instead of Java heap cache

• Memcached is a simple and efficient Unix daemon

• Several Java clients available

Only two parameters to set : memory size and listening port

All based on NIO !

Partitioned Memcached

Application Memcached

Memcached

Memcached

Memcached

MemcachedClient

Application

MemcachedClient

Application

MemcachedClient

R/W requests between the application and one of the memcached instances (depending on hashing)

Monitor everything

• A JMX attribute only costs an AtomicInteger and is priceless

• Spring JMX offers efficient annotations

• Hyperic can do the aggregating job

AtomicInteger doesn’t cost synchronization

But so awful to configure and use. SpringSource promises to makes it better !

@ManagedResource, @ManagedAttribute

Logging with care

• With high traffic, strange things happen

• These strange things may be hard to reproduce in development environment

• Logs are the only way to track them

Synchronization issues, connection losses, weird requests, ...

You’ll have a lot of logs to store, but it’s ok

Production environment’s behavior can’t be fully simulated

Concurrency Playground

What can be done with java.util.concurrent ?

• Parallel invocations, with or without dependencies between requests

• Making synchronous and asynchronous code collaboration possible

• Blocking IO code in pooled threads mixed with NIO code

Wrapping Future, CountDownLatch, NIO callbacks, ...

CountDownLatch, custom Future implementation, ...

ExecutorService with Future will do the job

Basic Parallel Requests

ServletThread

executorService.submit()

Thread A(from Pool)

Thread B(from Pool)

future.get()

future.get()

Basic Parallel Requests

ServletThread

executorService.submit()

Thread A(from Pool)

Thread B(from Pool)

Callable.call() Callable.call()

future.get()

future.get()

Basic Parallel Requests

ServletThread

executorService.submit()

Thread A(from Pool)

Thread B(from Pool)

Callable.call() Callable.call()

future.get()

future.get()

Thread Pooled Requests + Memcached NIO Client

ServletThread

Thread A(from Pool)

Thread B(from Pool)

get()

NIO Thread(Memcached)

invoke()(custom

ExecutorService)

future.get()

future.get()

Thread Pooled Requests + Memcached NIO Client

ServletThread

submit()(in read

callback)

Thread A(from Pool)

Thread B(from Pool)

get()

NIO Thread(Memcached)

invoke()(custom

ExecutorService)

future.get()

future.get()

Thread Pooled Requests + Memcached NIO Client

ServletThread

submit()(in read

callback)

Thread A(from Pool)

Thread B(from Pool)

call()

get()

NIO Thread(Memcached)

call()

invoke()(custom

ExecutorService)

future.get()

future.get()

Thread Pooled Requests + Memcached NIO Client

ServletThread

submit()(in read

callback)

Thread A(from Pool)

Thread B(from Pool)

call()

get()

NIO Thread(Memcached)

call()

invoke()(custom

ExecutorService)

future.get()

future.get()

Questions / Answers

?@mfiguiere

blog.xebia.fr