open ha cluster
TRANSCRIPT
Discovering Open High Availability Cluster
Robert Sohigian
Sun Microsystems
Outline
What is Open High Availability Cluster?
Why is High Availability Important?
How Solaris Cluster Provides High Availability
Cluster Agents
Cluster in Action
Open-Source Roadmap
How to Use Open HA Cluster
How to Participate
What is Open High Availability Cluster?
Open High Availability Cluster
The open-source code base for SolarisTM Cluster
Based on Solaris Cluster 3.2
Hosted by the HA Clusters community group on
opensolaris.org
http://opensolaris.org/os/community/ha-clusters/ohac/
Code available under the CDDL
Why is Sun Open-Sourcing Solaris Cluster?
Share
Public should have access to the source-base
We hope developers will contribute back their agents and other modifications
Open Development
Increase confidence in the cluster code-base
Enhance the code-base with third-party contributions
Why is Sun Open-Sourcing Solaris Cluster?
Shorten Feedback Loop
Easier to incorporate feedback
Find and fix bugs quicker
Early adoption of new features
Because Solaris Cluster is FUN!
Why Is High Availability Important?
Why is HA Important?
Downtime is costly
Failures are inevitable
Hardware, Software, Human Error, Disaster
With a single physical system, single points of failure are catastrophic
Network card dies, CPU misbehaves, disk drive crashes ...
HA Clusters automate the recovery process from inevitable failures, minimizing downtime and cost
Downtime is costy:
Example: Credit card database. If the DB is down, your card will be rejected. If your card is rejected, you'll use a different card instead of waiting for the DB to come back up. The bank loses your business.
Example: Ad-revenue-based web site. If your web server is down, you lose traffic and fail to generate ad revenue for that period of time.
Example: Online vendor. If your web server, business logic, or DB is down, your potential customers might go to a different site. You lose business.
Examples of failure: Hardware failure (CPU, memory, disk, network card, cable, ...), Software bug (application crash, OS panic, ...), Human error (Tripping over power cord, unplugging network cable, ...), Disaster (Earthquake, tornado, hurricane, terrorism, ...)
Why automated recovery: Operator Intervention is Annoying ( No one enjoys being on call 24/7), Operator Intervention is Costly (Overtime pay), Operator Intervention is Slow (Recovery usually slower than when automated)
Hardware Redundancy: A cluster of two or more physical systems can use hardware redundancy to recover from any single point of failure
Can use off-the-shelf hardware
How Solaris Cluster Provides High Availability
Open HA Cluster is the source-code for the Solaris Cluster product. In order to understand Open HA Cluster, we need to cover some background on Solaris Cluster.
Platform for High Availability
Tolerates Single Points of Failure (and some multiple failures)
Hardware redundancy with off-the-shelf hardware
Robust software HA infrastructure
Monitors the health of cluster infrastructure (hosts, shared storage, and network)
Orchestrates recovery of applications and cluster infrastructure (shared storage and IP networking)
Integrated tightly with the Solaris Operating System
Robust membership including quorum to prevent split-brain
Disk Fencing ensures Data Integrity in spite of failures
Solaris Cluster is a General purpose HA Platform for making Solaris applications Highly Available. It's a tightly-coupled distributed system of one or more physical computers. Supports both traditional failover and scalable HA
Product History
SC 3.2 FCS - Dec. 2006
SC 3.0 FCS - Dec. 2000, and several updates/releases in between
Prior products: HA 1.x 1990s, SC 2.y -1990s.
Hardware
Hardware Redundancy: Hosts, Dual private interconnects between hosts (Availability and bandwidth), Public (external to cluster) network: uses Solaris IPMP, Redundant NICs optional, IPMP can monitor single NIC, Storage connected to multiple hosts for availability, Volume manager for disk mirroring
File Systems: Cluster File System (PxFS), QFS for Oracle RAC configurations, ZFS, UFS, VXFS
Tightly integrated with Solaris
Allows the infrastructure to be resilient to load
Exploits kernel hooks for faster failure detection and recovery
Leads to higher and more predictable availability
Solaris Cluster Stack
OperatingSystemCluster InfrastructureAgentsApplications
Heartbeats
Membership
Spend a while on this slide explaining what a cluster is.
There are several (at least two) physical nodes, tightly coupled with software heartbeats.
Solaris Cluster is a layered product on top of Solaris.
Cluster Infrastructure (core) provides High Availability Platform for Applications
Applications run on cluster unmodified (off-the-shelf)
Cluster Agents (also called Data Services) are the glue layer between applications and cluster infrastructure
Custom agent for each application
Agent interacts with cluster core through APIs
Application cannot break into the high availability business on Solaris Cluster without an agent!
HA Failover Service
Application failover
Within nodes or container
Containers failover
Failover IP address
Failover Service
Service is hosted by a primary node in the cluster, with backup capability on one or more secondary nodes.
Exactly one service instance active at a time
Example: Database
Scalable ServiceSoftware Load Balancing
Global Network Service Provides Global IP address with failure protection
Scalable Service
Service is hosted by several nodes in the cluster at the same time, with backup capability on zero or more nodes.
Note that this diagram doesn't show all nodes directly connected to all other nodes, which is required for correct configuration.
Apache (Scalable)
Global Network Service Provides Global IP address with failure protection
MySQL (Failover)
Example: Apache and MySQL
Service is hosted by several nodes in the cluster at the same time, with backup capability on zero or more nodes.
Heartbeats
Membership
Quorum
Disk Fencing
Resource Group Manager
Resource (application) dependencies
Inter RG dependencies
RG affinities
HA Failover Service
Application failover
Within nodes or container
Containers failover
Failover IP address
Global File ServiceFailover File service
Scalable ServiceSoftware Load Balancing
Global Network Service Provides Global IP address with failure protection
MonitoringSolaris Cluster Architecture
Talk as little or much as folks are interested on this slide. Some more details:
Cluster Communications: Distributed Object Framework based on CORBA, Called the ORB, Available in kernel and userland; RPC and Doors
Cluster Algorithms: Heartbeats (Monitor nodes in the cluster over the private network, triggering reconfiguration when nodes join/leave, Resilient to load), Membership (Establishes clusterwide consistent cluster membership, Coordinates reconfiguration of other layers ), Quorum (Prevents partitions (split brain, amnesia) in the cluster, Protects against data corruption, Split brain is partition in space, such that cluster partitions can not talk to each other, but are up at the same time, Amnesia is partition in time, such that cluster partitions in different points in time are up, but the later one is not aware of the former and any changes committed to the cluster data by the former., Uses a majority voting scheme, 2 node clusters require a quorum device (an external tie-breaker), Disk Fencing (Used to preserve data integrity, Non cluster nodes are fenced off from updating any shared data)
Membership changes trigger algorithms of upper layers, including the ORB, CCR, Global File System or PxFS, Global Device Service, Global Networking, Resource Group Manager (in user space)
Resource Group Manager (RGM): Rich and extensible framework for plugging applications into Sun Cluster, RGM uses abstraction of a resource to represent an application, RGM interacts with the application Agent, which provides methods for controlling the application (Start, Stop, Monitor, Validate), Closely related resources placed in Resource Groups (Ex. HA-NFS: RG has 3 resources: NFS, IP, Storage), An RG is a basic failover unit, Supports both failover and scalable RGs , Support for rich dependencies between Resources and Rgs, Additional semantics for inter RG dependencies
Cluster Agents
Solaris Cluster Stack
OperatingSystemCluster InfrastructureAgentsApplications
Cluster Agents (Data Services)
Applications run on the cluster unmodified
Cluster agents are the glue layer between applications and cluster infrastructure
Custom agent for each application
Interacts with the cluster core through APIs
Provides start, stop, and other commands specific to the application to be called by the cluster framework
Provides monitor daemon specific to the application
Applications cannot break into the high availability business on Solaris Cluster without an agent!
Intentionally not getting into GDS or agent builder on this slide. That will come in the next slide. The point is to introduce the concept of an agent.Agents also called Data Services.
Agents Development
Several choices available
Sun Cluster Agent Builder
Generic Data Service (GDS)
Data Service Development Library (DSDL)
RGM Application Programming Interface (API)
In the context of making the application service highly available, we have made it easier by delivering predefined application service agents
These are in-house agents and agents developed by ISV themselves
Agents Available Include
Business Logic Tier
HA Sun Java System App Server PE/SE
HA Sun Java System App Server EE
HA Sun Java System Directory Server*
HA Agfa IMPAX
HA BEA Weblogic Server
Scalable Broadvision One-To-One*
HA IBM Websphere MQ
HA IBM Websphere MQ Integrator
IBM Lotus Notes+
HA Oracle Application Server
HA SAP liveCache
HA SAP J2EE Engine
HA SAP Enqueue Server
Scalable SAP
HA Siebel
HA SWIFTAlliance Access
HA SWIFTAlliance Gateway
HA Sybase Adaptative Server*
Web Tier / Presentation
HA Sun Java System Web Server
HA Sun Java System Messaging Server*
HA Sun Java System Message Queue
HA Sun Java System Calendar Server*
HA Sun Java System Instant Messaging
Server*
Scalable Sun Java System Web Server
HA Apache Web/Proxy Server
HA Apache Tomcat
Scalable Apache Web/Proxy Server
Database Tier
Oracle Parallel Server*
HA Oracle9i and Oracle 9i RAC*
HA Oracle 10g and Oracle 10gRAC*
HA Oracle E-business Suite
HA Oracle*
HA Sybase*
HA MySQL
HA SAP/MaxDB Database
HA PostgreSQL
* Not yet open-source
Management Infrastructure Tier
HA Sun N1 Grid Engine
HA Sun N1 Service Provisioning System
HA DNS, HA NFS
HA DHCP
IBM Tivoli+
Mainframe Rehosting (MTP)*
HA Samba
HA Solstice Backup*
HA Solaris Container
HA Symantec NetBackup*
Online Agent list:http://www.opensolaris.org/os/community/ha-clusters/ohac/Documentation/Agents/open-agents/
In the context of making the application service highly available, we have made it easier by delivering predefined application service agents
These are in-house agents and agents developed by ISV themselves
Not all these agents are open-source yet!!!
Open-Source Roadmap
Multiple Source Trees
Solaris Cluster consists of four different source trees
Agents
Core
Geographic Edition
SCATE
Since agents are open-source we'll focus on the agents code details (obviously we can't talk about the other code yet).
What's Open-Source Now
Most of the cluster agents
Geographic Edition
Cluster agents and geo edition G11N
CTI for TET test framework
Based on SCATE (Sun Cluster Automated Test Environment)
Used for both ON and Cluster tests
Test suites for agents and geo edition
More on SCATE:
A suite of automated tests and tools
Distributed test development framework
Distributed test execution framework
Client/server architecture
Easy to plug-in new test suites
Framework now available as CTI for TET (Common Test Infrastructure for Test Environment Toolkit)
Used for both ON and cluster test suites
SCATE Test Assets (not open-source yet)
50+ automated test suites, each with 100s of tests
500+ fault points in the product
350000+ functional tests, 45000+ faults injected
Much More to Come
2008 release source for
Sun Cluster core
Test suites for core
Asynchronously
Docs
Over three million lines of code to be open-sourced!
How to Use Open HA Cluster
In order to understand Open HA Cluster, we need to cover some background on the Solaris Cluster product.
Solaris Cluster Express
A complete version of Solaris Cluster software that runs on Solaris Express
Solaris Cluster Express 02/08 (current release)
Runs on Solaris Express Developer Edition 1/08
Runs on SPARC and 32 or 64-bit x86 hardware
Support available only through community forums
Where to get it
http://opensolaris.org/os/community/ha-clusters/ohac/downloads/
Solaris Cluster 3.2
Most recent release of the Solaris Cluster product
Runs on Solaris 9 and 10
Supported on SPARC and 64-bit x86 hardware
Try it out for free
Where to get it
http://sun.com/software/solaris/cluster/get.html
Build and Run Agents and Geographic Edition
Install
Solaris Express Developer or Community Edition
Compiler
Sun Studio 11, Sun Studio 12, or gcc
Build tools
ON build tools (SUNWonbld)
OHAC build tools (SUNWscbld)
Download source tarballs and archives
One-line nbuild command
Install packages on Solaris Cluster Express or Solaris Cluster 3.2
Build, Install, and Run OHACTS
Download CTI for TET source from testing community
http://opensolaris.org/os/community/testing/testsuites/ctifortet/
Follow CTI for TET instructions to build it
Download OHACTS source
http://opensolaris.org/os/community/ha-clusters/ohac/downloads/
Build and install OHACTS package http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/Tests/buildntest/
Run OHACTS
http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/OHACTS.pdf
How to Participate
Where to Start
HA Clusters Community Group Web Page http://www.opensolaris.org/os/community/ha-clusters/
Open HA Cluster page http://opensolaris.org/os/community/ha-clusters/ohac/
[email protected] mailing listhttp://mail.opensolaris.org/mailman/listinfo/ha-clusters-discusshttp://opensolaris.org/jive/forum.jspa?forumID=195
How to Contribute to Open HA Cluster http://opensolaris.org/os/community/ha-clusters/ohac/contribute/
What You Can Do
Browse the code with OpenGrok
http://src.opensolaris.org/source/xref/ohac/
Join the discussions on ha-clusters-discuss.
File bugs
http://www.opensolaris.org/bug/report.jspa
Fix a bug
http://opensolaris.org/os/bug_reports/oss_bite_size/
Contribute to an existing project
http://www.opensolaris.org/os/projects/
Start a new project
Two new community projects created
HA-Informix
Cluster Agent for Informix Dynamic Server
Created 1st August 2007
Design Doc, Cheatsheet etc. available
http://opensolaris.org/os/project/ha-informix/
HA-xVM
Cluster Agent for Solaris xVM guest domains
http://opensolaris.org/os/project/ha-xvm/
Possible Projects
Develop Agents for
Adabas, BMC Control-M, BMC Patrol Server, BMC Performance Manager, CA Unicenter, Citrix, Connect Direct, EMC Documentum, FileNet, IBM Tivoli Client, IBM Tivoli Server, IBM Tivoli Workload Scheduler, IBM Websphere Application Server, Informatica, Java CAPS, Java DB, JBOSS, Netbackup, Netbackup Client, Open LDAP, Sleepy cat DB, Sun Java System Identity Server, Sun Java System Portal Server, Sybase Replicator agent, TSM Backup client, UC4
More project suggestions for Open HA Cluster http://opensolaris.org/os/community/ha-clusters/ohac/contribute/agents-suggested-projects/
Or propose your own project!
Thank you!
Robert [email protected]
Based on Slides by
[email protected]
open artwork and icons by chandan:
http://blogs.sun.com/chandan
Click to edit the title text format
Click to edit the outline text format
Second Outline Level
Third Outline Level
Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
Ninth Outline Level
USE
IMPROVE
EVANGELIZE
Click to edit the notes format
Click to edit the title text format
Click to edit the outline text format
Second Outline Level
Third Outline Level
Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
Ninth Outline Level
USE
IMPROVE
EVANGELIZE
Click to edit the notes format
Click to edit the outline text format
Second Outline Level
Third Outline Level
Fourth Outline Level
Fifth Outline Level
Sixth Outline Level
Seventh Outline Level
Eighth Outline Level
Ninth Outline Level
USE
IMPROVE
EVANGELIZE
Click to edit the notes format