ARCHITECTING
EXTREMELY LARGE
SCALE WEB
APPLICATIONS A MUST read for every architect
ABSTRACT An overview of the much needed vicissitude in
the architectural thought transformation from
monolithic to microservices architecture.
PRASHANTH B PANDURANGA Director of Technology
New York times auto scaled to 500,000 users, HipChat has about 1.2 billion messages/documents stored,
Sales force deals with 1,300,000,000 daily transactions with over 24,000 database transactions per second
and over 22PB of raw SAN storage capacity, CinchCast has over 50 million page views a month, Pinterest
has over 18 million visitors with a 10X growth rate, Amazon has over 55 million active customer accounts,
Flickr has over 4 billion queries per day, Netflix has 48 million members with over 50,000 requests per
second and the list goes on. . Thanks to HighScalability, where the above statistics are derived from, you
get a picture of the websites and the scale that I am referring to!
I had covered the SAAS requirements in my previous blog. . In this blog I shall provide a bird’s eye
view of the technologies used by a few large scale websites and their impact on the multi-tier web
application architecture.
Reference: HighScalability has some awesome architecture blogs, I have derived the technology stack from
multitude of articles hosted by highscalability and a few from Netflix blogs
http://highscalability.com/blog/category/example
Technology Stack
This section is not meant to be an extensive overview of each of the below websites and their technology
stack, but rather provides an overview of the variety of tools used and the logical layers that form the
architecture.
NOTE: The below mentioned tools used by the companies are point in time and may have been changed.
Figure 1 NewYorkTimes Technology Stack
Figure 2 HipChat Technology stack
Figure 3 Salesforce Technology stack
Figure 4 Netflix Technology stack
I have elaborated on SAAS requirements and general architectural requirements in my previous blog.
https://prashanthpanduranga.wordpress.com/2015/03/25/inevitability-of-multi-tenancy-saas-in-product-
engineering/
The requirements which apply the most to large scale web applications:
Performance: There are lot of statistics published relating to performance implications of web applications.
One such statistic, Users abandon website even if there is a 2 second delay during a transaction. Large scale
web applications obviously have very high performance requirements. A very large percentage of
applications gets redesigned primarily for a better user experience. Average Page Load Time as a fraction
of Server and client time, Network time, Page views, Bounce Rate, Percentage Exit, Average Redirection
Time, Average Domain Lookup Time, Average Server Connection Time, Average Server Response Time,
Average Page Download Time, Average content load time, Average session time, DNS resolution time, TCP
connection time, Time to first byte, Full page object load time, Requests per second, error rates, Peak
response time, Uptime, CPU utilization, Memory Utilization are just a few metrics to look for
Availability: Business continuity is of utmost importance. Various availability techniques can be applied on
every layer. AlwaysOn Failover Cluster Instances, AlwaysOn Availability Groups, Database mirroring, Log
shipping, Redundancy models - Active-Active, Active-Passive, Redundancy Methods - Hot Standby, Warm
Standby, Cold Standby, and the measurement of the same expressed as mean time to failure, mean time
to repair, Availability as a measure of MTTF / (MTTF + MTTR), Eliminating single points of failure,
Accelerating fault detection, isolation and resolution, hot spares, warm spares, cold spares, clustering,
RAID, redundancy, Heart beats, watermarking resources, check pointing, Watch dogs and more
Monitoring and Diagnostics:
26 front end proxy serves. Double that in backend app servers [Atlassian HipChat], 15,000+ hardware
systems – [Salesforce], 100 hardware nodes in production – [CinchCast], 180 Web Engines + 240 API
Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, 200 Memcache Instances, 4 Redis
Task Manager + 80 Task Processors, Sharded Solr – [P interest], 1000+ supported devices – [Netflix]
Imagine maintaining those, when there are innumerable servers involved, failure of system components is
common, needless to say, to take appropriate timely actions monitoring and diagnostics plays a very
important role.
Scalability: Capability of supporting and optimizing resource utilization on increasing workloads on various
dimensions such as memory, cores, data structures, throughout and more. Goes without doubt that the
application need to scale, in order for the application to perform well, and without automating it, the
application cannot stay inexpensive.
Automation: Identifying failures and automating re-provisioning of those components/servers is extremely
important
Architecture has significantly emerged from a monolithic architecture to microservices architecture.
Figure 5 Minimalistic layers
Let’s take a look at the logical segregation of layers in current large scale web applications.
APPLICATION
DATA
PRESENTATION
SERVICE/BUSINESS LOGIC
DATA
In the following table we look at some of the technologies used by high scale websites and compare them
to other similar tools/technologies
Note: The technologies covered by the above websites was used as the benchmark for comparison.
Comments are provided only where I wanted to provide additional details.
Tools/software
Usage/Description
TOP few similar tools/software to consider
Comments
RabbitMQ Message broker system Message oriented middleware Queuing software ESB
ActiveMQ, Amazon SQS, HornetQ, HiveMQ, JMS, Kafka, ZeroMQ, MSMQ, NServiceBus, Azure Service Bus, OpenMQ, Redis, Storm, Akka, Apache Camel and Spring, OFM, Fuse ESB, WebsphereMQ, Windows Service Bus, BizTalk, WSO2, Mule, Talend ESB, Gearman, JBoss, ServiceMix, OpenESB, Apache QPID
Look out for AMQP compliance Some of the tools referred aren’t Message brokers but are used in conjunction to perform the same Other AMQP PIKA Shovel
Kinesis Real time data processing
Kafka, Storm
Tornado Web Server HTTP Server
Nginx, Apache, IIS, lighttpd, haproxy, varnish, glassfish, Jetty, Geronimo, Tomcat,
Some are even used as reverse proxy, proxies,
PRESENTATION
SERVICE/BUSINESS LOGIC
DATA
SEC
UR
ITY
AN
ALY
TIC
S
MO
NIT
OR
ING
& D
IAG
NO
STIC
S
STORAGE
EVEN
TS
NO
TIFI
CA
TIO
N
VIRTUALIZATION
NETWORKCOMPUTE
HARDWARE LAYER
CLOUD
CLOUD ADAPTER
AU
TOM
ATI
ON
/BA
TCH
INTEGRATION
PROCESSING ENGINE
CACHEM
AN
AG
EMEN
T
AD
-HO
C L
AYE
R
AU
DIT
ING
CO
NFI
GU
RA
TIO
N
DISTRIBUTED PROCESSING
INGESTION
DATA MANAGEMENT DATA RULES META DATA DATA QUALITY
CLICKSTREAM DATA SOURCES
CHAT
SENSOR
SOCIAL
LOGS
CRM
ERP
APPLICATION DATA
CHANNELS
EDW
METADATA MANAGEMENT MULTI TENANT PROCESSING PARALLEL COMPUTE
PARALLEL OPERATIONS COMPLEX EVENT PROCESSING GOVERNANCE
WORKFLOW
RE
PO
RT
ING
STREAM PROCESSING
IN-MEMORY PROCESSING
MESSAGE TRANSPORTSMESSAGE QUEUESMESSAGE BROKERS
INTEGRATION FRAMEWORKSENTERPRISE SERVICE BUS
INTEGRATION SUITES
OPERATING SYSTEM
Cassandra Distributed database management system
Mongodb, aerospike, accumulo, azure table storage, bigtable, couchbase, couchdb, dynamodb, datastax, ElasticSearch, Greenplum, Vertica, HBase, InfiniDb, InnoDB, MariaDB, neo4j, Netezza, TeraData, RedShift, Riak, RavenDB, Solr, Spark, VoltDB,
With over 200 dbs, it’s difficult to list all: Checkout the below link https://prashanthpanduranga.wordpress.com/2013/12/23/why-nosql-ok-but-why-so-many/ Some of them are dataware housing Solutions, While some are data processing engines
Hana In-Memory database
GemFire, Hekaton, Aerospike, BigMemory, DataBlitz, EhCache, eXtremeDB, FuelDB, HazelCast, MonetDB, Coherence, VoltDB
Any Key value store can be used for the same, some of the enterprises have experimented using NoSQL store used as a cache and unstructured db solution
Linux Operating System SUSE, FreeBSD, Solaris, Debian/Ubuntu, Windows Server, Mac OS X, RHEL
Only Server OS included in the list
SockJS Web Socket like object
Web socket, socket.io, atmosphere, SignalR, Alchemy, Fleck
Couple of them listed are open source
Libev Event Loop LibEvent, Asio, Nginx, epoll Fabrik Visual
programming IDE Also known for Content construction Kits (CCK)
IDE: Visual Studio, Eclipse, NetBeans, Aptana CCKS: Seblod, K2, chronoform, Zoo, Breezingforms, Cobalt, FlexiContent
Java Programming language
C, C++, Python, C#, PHP, Javascript, Ruby, R, Matlab, Objective-C, Visual Basic, Perl, Swift, Scala, Shell, GO, LISP, SAS, F#, Groovy, Lua
Some of them listed are web programming languages adiitionals: HTML, SQL, Haskell
Twisted Event driven network programming framework
Tornado, Django, Asyncio,
AWS Cloud provider Azure, Rackspace, CenturyLink, Salesforce, Engineyard, Google, OpenStack, SAP, CloudBees, CumuLogic, Eucalyptus, Gigaspaces, Mulesoft, Parallels, Pivotal, puppet Labs, Ravello,
The list includes infrastructure, platform, storage and security cloud providers
Rightscale, SoftwareAG, Xively, AT & T, Cisco, Comcast, EMC, GoGrid, CSC, HP, IBM smartcloud, Joyent,
Lucene Test search Engine Library
Azure Search, Autonomy, Solr, GSA, Attivio, DTSearch, elasticSearch, endeca, FAST, MarkLogic, Nutch, Sphinx, Sketchy, Scumblr
A few NOSQL databases have been used for the same, This list does not include all the NOSQL databases that could be used
Adobe Air Cross-platform runtime
Cordova(Phonegap), Appcelerator, Qt, Sencha, cocos2d-x, Xamarin, ionic, Kony, mono, xcode
The ones listed here are cross platform as well as mobile development platforms.
Sensu Monitoring Framework
Zabbix, Nagios, icinga, monit, Riemann, statsd, graphite, zenoss, collectd, munin, cacti, new Relic, ganglia, splunk, sentry, dynatrace, datadog, skylight, zenoss, observium, spiceworks, solarwinds, fiddler, wireshark, httpwatch, firebug, soapUI, OpManager
The list includes some of the: Infrastructure monitoring Searching, monitoring and analysing, Network monitoring Scalable distributed monitoring system
PagerDuty NeoLoad
Incident management system and performance testing and monitoring
OpsGenie, VictorOps, xmatters, pingdom, Gomez, webpagetest, monitis, uptrends, keynote, OpsView, Apache JMeter, LoadRunner, WebLOAD, Appvance, NeoLoad, LoadUI, WAPT, Loadster, LoadImpact, Soasta, Rational Performance Tester, Testing Anywhere,OpenSTA, QEngine (ManageEngine), Loadstorm, CloudTest, Httperf, SilkPerformer, BlazeMeter, Visual Studio Test Suite,
Also includes web site monitoring Cloud based quality testing Performance monitoring
Chef IT Automation Puppet, ansible, salt, docker, Jenkins, Capistrano, saltstack
Configuration management, SCCM
memCache Distributed memory object caching
Apc, memcached, dynacache, ehcache, xcache,
key value based NOSQL databases are also used
Razor Physical and virtual hardware provisioning solution
Axemblr, Cobbler, JuJU, SaltCloud, Dell Crowbar, Ansible, CFEngine, Chef
Perforce Version Management and Content collaboration
Git, SVN, TFS, bitbucket, ClearCase, Subversion
Pytheas ITIL assets management software
Remedy (BMC), Assyst (Axios), FrontRange, EasyVista, Hornbill, HP Service Manager, SmartCloud Control Desk (IBM), ServiceNow
IT incident management, IT problem management, IT change management, IT release governance, IT user self-service, IT request management, IT knowledge management, IT service support analytics and reporting, IT SLA management Ref: Gartner
ZUUL Service that provides dynamic routing, monitoring, resiliency and security
Nginx, lightpd, Netscaler, HAProxy, Radware, CoyotePoint, Barracuda, Kemp, Varnish, Avast, Norton, Kaspersky, Mcafee, AVG, Avast, Bitdefender, F5, PaloAlto, Cisco ASA, Cisco ACE, Foundary, Juniper SSG, MS TMG
Can be firewall,
router, web load
balancing server,
proxy Server etc.
Feign Java http client binder
Retrofit, JAX-RS, web socket, Jersey, CXF, Apache HC
Includes transport libraries
Hive Querying and managing large datasets residing in distributed storage
Impala, BigSQL, HAWQ
AWS ELB Elastic Load balancing
Nginx, HAProxy, Route53, Azure Traffic Manager, F5
Port-bound servers,
sticky sessions, TCP
session
reassignment,
automatic unfail,
slow start,
SynGuard, dynamic
feedback protocol,
NAT, maximum
connection, Round
Robin, Least
Connections,
Weighted Round
Robin, Weighted
Least Connections,
Fastest Response Layer 4 and Layer 7 load balancing Cloud Load balancing features: Dedicated (static) IP address,SSL termination Multiple protocols, Advanced access control, Connection logging, Advanced algorithmic routing, Session persistence, Connection throttling, Node management, High availability Content caching, Persistent connections, Gzip compression, Regionalized load balancers
gZip Application used for file compression and decompression
httpZip,deflate, 7zip, bzip2, zlib
Akamai Content delivery network
Azure CDN, Cloudfront, Torbit, Incapsula, Cotendo, Fastly
HTML 5 frameworks Javascript Frameworks
https://www.facebook.com/notes/prashanth-panduranga/frameworks/10152107517972934
OpenStack Open Source Cloud computing platform
OpenStack currently has the following features: Compute(Nova), Object Storate (Swift), Block Storage (Cinder), Networking (Neutron), Dashboard (Horizon), Identity Service (Keystone), Image Service (Glance), Telemetry (Ceilometer), Orchestration (Heat), Database (Trove), Bare Metal Provisioning (Ironic), Multiple tenant cloud messaging (Zaqar), Elastic Map Reduce (Sahara)
Hadoop Distributed storage and distributed processing of very large data sets on computer clusters
Aegisthus Bulk Data Pipeline out of Cassandra
Eureka Eureka is a REST (Representational State Transfer) based service that is primarily used in the AWS cloud for locating services for the purpose of load balancing and failover of middle-tier servers
Genie Federated Job Execution Engine
Clojure Dynamic programming language that targets the Java Virtual Machine PigPen Map-Reduce for Clojure
Governator Governator is a library of extensions and utilities that enhance Google Guice to provide: classpath scanning and automatic binding, lifecycle management, configuration to field mapping, field validation and parallelized object warmup
Inviso Visualize Hadoop performance
Ribbon Ribbon is a Inter Process Communication (remote procedure calls) library with built in software load balancers
Hystrix Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable
Suro Distributed data pipeline Aminator A tool for creating EBS AMIs
Lipstick Pig Visualization framework Zeno In-Memory Data Propagation Framework
Blesk Lightweight client for pushing notifications to web based applications/sites Turbine Turbine is a tool for aggregating streams of Server-Sent Event (SSE) JSON data into a
single stream. The targeted use case is metrics streams from instances in an SOA being aggregated for dashboards
Priam Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra
Workflowable Workflowable is a Ruby gem that allows adding flexible workflow functionality to Ruby on Rails Applications
s3mper S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index
Astyanax Java Client for Apache Cassandra
Denominator Denominator is a portable Java library for manipulating DNS clouds. Denominator has pluggable back-ends, including AWS Route53, Neustar Ultra, DynECT, Rackspace Cloud DNS, OpenStack Designate, and a mock for testing
GCViz Garbage Collector Visualization framework
Curator The Curator Framework is a high-level API that greatly simplifies using ZooKeeper. It adds many features that build on ZooKeeper and handles the complexity of managing connections to the ZooKeeper cluster and retrying operations
Staash A language-agnostic as well as storage-agnostic web interface for storing data into persistent storage systems, the metadata layer abstracts a lot of storage details and the pattern automation APIs take care of automating common data access patterns
Edda Edda is a Service to track changes in cloud deployments
Brutal An asyc centered chat bot framework for python programmers written using the twisted framework
CassJMeter JMeter plugin to run cassandra tests Glisten Groovy library for building JVM applications with Amazon Simple Workflow (SWF)
Pig Platform for analyzing large data sets Spark Engine for big data processing, with built-in modules for streaming, SQL, machine
learning and graph processing
Karyon Framework and a library for a cloud ready web service. Blueprint for the services. It contains Bootstrapping, Libraries and Lifecycle Management, Runtime Insights and Diagnostics, Pluggable Web Resources, Cloud-Ready hooks
EBS Elastic Block store, persistent block level storage volume
Curler A Gearman worker which cURLs to do work archaius, , Library for configuration management API
ZooKeeper ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services
Parallel processing - Explicit and Implicit parallelism, batch parallelism, asynchronous programming,
segregating layers, distributing workloads, Load balancing, multi- tenancy, scaling out on all layers,
sharding, partitioning, CAP preference, reads, writes, statelessness, logging and telemetry, automating,
SOA adoption, caching, throttling, distributing requests across multiple zones, effective usage of CDNs,
Auto provisioning, Auto scaling, compression, queuing, workload distribution, batch processing, designing
system with fault tolerance, redundancy, Consistency, Availability, Partition Tolerance, event processing,
web sockets, cloud computing, fog computing, Grid Computing, Client side workload distribution, In-
Memory processing, Proxies, No single points of failure. Resilience to failure, Graceful degradation,
Recoverability from failure, design for failure, Database Transactions, Client side transactions, two-phase
commit, Auto-commit, Partition Everything, DB operations ordering, Considerations for Eventual
consistency, Functional Segmentation, Application Pools, Prevention of session state, Async Everywhere,
Index, Structured Indexes, text indexes, entity indexes, Fuzzy match indexes, pre-aggregated indexes, pre-
calculated indexes, embedded value indexes, join indexes, link indexes, De-Normalized Indexes (all kinds)
are all important considerations for a highly successful and scalable website.
Rest assured if you have considered all the above factors in your architecture you are on your way to create
a scalable one. Do let me know if you have questions regarding any particular subject and I will be glad to
write up on the same. .