hybrid cluster technology whitepaper

8/8/2019 Hybrid Cluster Technology Whitepaper

1/7

Hybrid Web Cluster Technology Overview

A brief introduction to the technologies in Hybrid Web Cluster.

Hybrid Web Cluster

Technology OverviewA brief introduction to the technologies

present in Hybrid Web Cluster.

Document version: 1.0Date: 5th August 2010

http://www.hybrid-cluster.com/

Page 1 of 7
http://www.hybrid-cluster.com/http://www.hybrid-cluster.com/


2/7



Table of Contents

Hybrid Web Cluster Technology Overview.........................................1System Overview.................................................................................3Redundancy..........................................................................................4Scalability.............................................................................................5Management Tools...............................................................................6Contact Information.............................................................................7



Page 2 of 7


3/7



System OverviewAt the heart of each Hybrid Web Cluster in-

stallation is an advanced system administrator

which automatically manages the cluster, re-

sponding to hardware or network issues auto-

matically and within seconds so that yourwebsites stay online .Each cluster can be comprised of individual

nodes of either real physical servers or virtualcloud server instances. New cloud server in-

stances can be automatically brought online

by the software to cope with peaks in de-

mand. Websites are automatically and trans-

parently load balanced around the cluster

such that no single server is ever overloaded

and near-live backups are stored on an administrator-configurable number of servers.

This means that as your hosting requirements grow in terms of storage capacity, performance

management and fault tolerance, you dont need to think about how to scale either vertically

or horizontally. Weve done the thinking for you.

C omplete fault tolerance means that if a hard disk,

a server, or even an entire data centre fails, the

software will recognise this, restore data from a

backup no more than 15 seconds old and continue

serving websites as if nothing had happened.

Hybrid Web Cluster is designed to run on FreeBSD

and provide the standard Apache, MySQL and PHP

stack, so normal LAMP applications will run

unmodified and scale from zero resource usage to

requiring a dedicated web and database server allto themselves automatically and instantly.

Through the use of multi-master database replication the system is able to scale beyond this

limit provided that users modify their code to only store variable state data in the database.



Page 3 of 7
http://www.hybrid-cluster.com/scalability/http://www.hybrid-cluster.com/scalability/http://www.hybrid-cluster.com/redundancy/http://www.hybrid-cluster.com/http://www.hybrid-cluster.com/redundancy/http://www.hybrid-cluster.com/http://www.hybrid-cluster.com/scalability/http://www.hybrid-cluster.com/scalability/


4/7



RedundancyIn order to achieve military-grade fault tolerance,

Hybrid Web Cluster allows you to specify exactly

how many redundant copies of each website are

kept starting at 2 for basic business redundancy.

This means that each website will be stored on

two servers at any one time, so if either one of

them fails you will not experience any downtime.

However, the redundancy level (or redundancyinvariant) is fully configurable and limited only

by the number of available cluster nodes you

could elect to have each website continuously backed-up to every single node in the cluster.

Hybrid Web Cluster has no single point of failure. The redundancy invariant guarantees that

data is sufficiently replicated at all times to cope with the failure of many nodes, and upon

failure the nodes act quickly to re-establish the redundancy. In other words, your data is always

safe, and failures are dealt with quickly and automatically.

Hybrid Web Cluster makes use of the ability of the

ZFS filesystem to take a snapshot of a filesystemand then send only the changes that have been

made over the network to another machine and

have the filesystem rebuilt. Each website or

database is divided onto its own ZFS filesystem,

and these filesystems are monitored for changes

when a change happens (at most every 15

seconds), our filesystem daemon creates a

snapshot of the filesystem and sends it to each of

the other cluster nodes that hold a backup copy of that particular website or database.

This means that when a failure occurs, a snapshot of each website on the failed server should be

available with data no more than 15 seconds old. The cluster uses this data to automatically and

instantly restore service with seconds of a failure rather than hours.



Page 4 of 7


5/7



ScalabilityBecause Hybrid Web Cluster can run in a cloud

computing environment, the software can decide to

automatically spin-up new servers from the

underlying cloud providers infrastructure, if it sees

that a particular website needs to scale. The cluster

will reconfigure itself within seconds of a spike in

traffic so that two big servers (one web, one

database) are dedicated to hosting your popularwebsite, and the other servers carry on sharing all your

other sites between them.

The fact that the cluster can dynamically change its

own size and rapidly cope with changes in demand in

this way is all down to Hybrid Web Cluster's unique

'Site-Juggling' load balancing algorithm.

Site Juggling works by transparently moving websites

between servers to cope with changes in demand.

When a new node is added to the cluster, the rest ofthe cluster reacts by gradually moving over some of

the websites it's hosting until the load is evenly

balanced again.

Responsible for this load balancing and dynamic sizing of the cluster is the StateMachine this

is our name for our home-grown cluster management daemon. This program runs on each

cluster node and monitors the load on each website or database, decisions about when to move

a website from one node to another are made on the nodes themselves based on global state

information which is broadcast over Spread.

The limitation of this approach is that the smallest unit of work is either a single website orsingle database this means that whilst maintaining compatibility with existing LAMP web

applications, the system cannot scale beyond a single website requiring its own separate

dedicated web and database servers. In order to scale beyond this limitation we use multi-

master database technology, although this requires applications be modified to only store

variable state data in the database.



Page 5 of 7
http://www.hybrid-cluster.com/cloud-computing/http://www.hybrid-cluster.com/cloud-computing/http://www.hybrid-cluster.com/http://www.hybrid-cluster.com/cloud-computing/http://www.hybrid-cluster.com/cloud-computing/http://www.hybrid-cluster.com/cloud-computing/http://www.hybrid-cluster.com/


6/7



Management ToolsAt the core of Hybrid Web Cluster is a

web hosting control panel responsible

for adding sites and databases,

managing e-mail settings, automatic

installation ofcommon applications

such as WordPress. The control panel

offers a single point of access allowing

you to make configuration changesaffecting the whole cluster in what

appears to be one centralised location

(of course the control panel is

resistant to failure as any website

hosted on the cluster would be).

Although Hybrid Web Cluster itself is almost completely self-managing, it is still useful and

interesting to be able to see whats going on in the cluster, live, as it happens. Within the

cluster administration control panel, a powerful set of visual representations of the cluster

allow you to see which servers are hosting which websites, how much load each individual node

or website is experiencing at any given time, as well as allowing you to actually watch as

websites are sitejuggledaround the cluster.

The web cluster control panel is a modern

PHP web application using jQueryUI and

AJAX to provide a responsive and easy-to-

use interface, all changes to the cluster are

made via StateMachine, or YakD (the

automated system administration daemon)

these two programs together are

responsible for rolling out configuration

changes to cluster nodes and keeping

configurations up-to-date as websites are

juggled around the cluster.



Page 6 of 7
http://www.hybrid-cluster.com/application-support-list/http://www.hybrid-cluster.com/application-support-list/http://www.hybrid-cluster.com/load-balancing/http://www.hybrid-cluster.com/http://www.hybrid-cluster.com/application-support-list/http://www.hybrid-cluster.com/load-balancing/http://www.hybrid-cluster.com/


7/7



Contact InformationMike Smithson

+44 (0)845 508 1116

http://www.hybrid-cluster.com/contact/

[email protected]



Page 7 of 7
http://www.hybrid-cluster.com/contact/mailto:[email protected]://www.hybrid-cluster.com/http://www.hybrid-cluster.com/contact/mailto:[email protected]://www.hybrid-cluster.com/

hybrid cluster technology whitepaper

Documents