ha cluster with opensuse leap

High Availability Cluster with openSUSE Leap

M. Edwin Zakariamedwin@opensuse.org

2

• Mohammad Edwin Zakaria• Linux user since 1998• openSUSE since 6.2 around 1999

https://en.opensuse.org/User:Medwin• openSUSE member

https://connect.opensuse.org/show/Medwin• openSUSE Indonesia

3

4

5

6

7

8

What is

- Cluster ?

- High Availability ?

9

Curious?

• A computer cluster consists of a set of loosely or tightly connected computers that work together so that, in many respects, they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software

10

Curious?

• High availability (HA) is a system that is designed to avoid the loss of service by reducing or managing failures as well as minimizing planned downtime for the system. We expect a service to be highly available when life, health, and well-being, including the economic well-being of a company, depend on it.

11

Curious?

• Harvard Research Group divide the HA into several Availability Environment Classification (AEC): AE4, AE3, AE2, AE1, AE0

• http://www.hrgresearch.com/pdf/AEC%20Defintions.pdf

• Other categories: continuous availability, fault tolerance, disaster tolerance

12

Once again what is cluster?

• High performance computing• Load balancer (high capacity)• High availability

‒ 99.99%

‒ MTBF (mean time between failure = total operating time/total numbers of failure)

‒ Single point of failure

13

Once again what is cluster?

14

Challange in HA

• Murphy’s Law “If anything can go wrong, it will”‒ Loss of data

‒ Service outage

• Flood, fire, earthquake, natural disaster, hardware damage‒ Can you afford a downtime?

‒ Can you afford low availability system?

‒ Cost of downtime?

15

Different between HA term

• HA term is widely use• VMware vSphere HA

‒ Closed source

‒ Hypervisor level and host hardware level

• openSUSE/SUSE HA‒ Open source

‒ OS level

‒ Protect critical resources running on VM

‒ HA within Linux OS

16

HA in Linux • Started with heartbeat project in around 1990• Now manage by ClusterLabs http://clusterlabs.org/• The ClusterLabs stack, incorporating Corosync and

Pacemaker defines an Open Source, High Availability cluster offering suitable for both small and large deployments.

• Pacemaker has been around since 2004 and is primarily a collaborative effort between Red Hat and SUSE, they also receive considerable help and support from the folks at LinBit and the community in general.

17

Hardware Consideration• External network, high traffic, use FO or eth

bonding• Communication network between cluster

node, use for messaging, membership, STONITH

• Storage network, use FO or eth bonding• Manage switch• STONITH/fencing device• Shared storage: NAS (nfs/cifs), SAN (fc/iscsi)

18

Hardware Consideration

19

Hardware Consideration

20

Software Component• Corosync

‒ messaging and membership

• Pacemaker‒ Cluster resource management

• Resource Agents‒ Manage and monitor availability of service

• Fencing device‒ STONITH to ensure data integrity

• User interface‒ Crmsh and Hawk

21

Other component• LVS linux virtual server• HAproxy• Shared file system: OCFS2, GFS2• Block device replication: DRBD• Shared storage: SAN• Geo cluster

22

More details

• Pacemaker :

Pacemaker is a cluster resource manager. It achieves maximum availability for your cluster resources by detecting and recovering from node and resource-level failures by making use of the messaging and membership capabilities provided by your preferred cluster infrastructure (either Corosync or Heartbeat).

23

More details

• Corosync :‒ provides cluster infrastructure functionality

‒ provides messaging and membership functionality

‒ maintains the quorum information.

‒ This feature has been utilized by pacemaker to provide high availability solution.

24

In short ...

• Corosync : A quorum system that notifies applications when quorum is achieved or lost

• Pacemaker :‒ To start/stop resources on a node according to the

score.

‒ To monitor resources according to interval.

‒ To restart resources if monitor fails.

‒ To fence/STONITH a node if stop operation fails.

25

Pacemaker Corosync Conceptual Overview

26

Pacemaker Components

• Non-cluster aware components (illustrated in green). These pieces include the resources themselves, scripts that start, stop and monitor them

• Cluster Resource manager, provides the brain that processes and reacts to events regarding the cluster

• Low level infrastructure, Corosync provides reliable messaging, membership and quorum information about the cluster

27

Pacemaker Stack• pacemaker corosync cluster called as pacemaker stack

• Linux kernel by default comes with DLM (distributed lock manager). It provides locking feature which will be used by cluster aware filesystem

• The GFS2 (Global File System2) and OCFS2 (Oracle cluster File System 2) are called as cluster aware filesystem

• To access single filesytem by multiple hosts you need to have either GFS2 or OCFS2.

• Or you can create a file system on top of cLVM (cluster logical volume manager)

28

29

Cluster Filesystem• If you have shared disk and want several nodes access

it, you need cluster aware filesystem

• The open source solution are GFS2 (Global File System2) and OCFS2 (Oracle cluster File System 2)

30

Cluster Block Device• DRBD (distributed replicated block device) allows you

to create a mirror of two block devices that are located at two different sites across an IP network. When used with Corosync, DRBD supports distributed high-availability Linux clusters. It is a network based raid1, and high performance data replication over network

• CLVM2, see https://www.sourceware.org/lvm2/

• Cluster md raid1, see https://www.kernel.org/doc/Documentation/md-cluster.txt

31

Cluster Block Device

32

STONITH• STONITH is an acronym for “Shoot-The-Other-Node-In-

The-Head”.

• It protects your data from being corrupted by rogue nodes or concurrent access.

• Just because a node is unresponsive, this doesn’t mean it isn’t accessing your data. The only way to be 100% sure that your data is safe, is to use STONITH so we can be certain that the node is truly offline, before allowing the data to be accessed from another node.

• STONITH also has a role to play in the event that a clustered service cannot be stopped. In this case, the cluster uses STONITH to force the whole node offline, thereby making it safe to start the service elsewhere.

33

Split brain – the HA problem• Two nodes run the same service, break the

data integrity

• Solution:‒ Quorum

If cluster doesn’t have quorum no action will be taken, means fencing and resource management are disabled without quorum

‒ STONITH

Shoot the other node in the head

• More on stonith http://ourobengr.com/ha/

34

Reference• SUSE HA Extension Doc (can be use for openSUSE

also) https://www.suse.com/documentation/sle-ha-12/

• HA clusterlabs http://clusterlabs.org

• Corosync doc http://landley.net/kdocs/ols/2008/ols2008v1-pages-85-100.pdf

• DRBD http://drbd.linbit.org/en/

• OCFS2 https://ocfs2.wiki.kernel.org/

• CLVM2 https://sourceware.org/lvm2/

• Linux SCSI http://linux-iscsi.org/

Case Study / Hands-on

36

Setting up HA on Leap

• Scenario: ‒ setting up openSUSE Leap 42.1 as host

‒ create 2 VM with QEMU/KVM, install openSUSE Leap 42.1, configure the network, and all the required packages

‒ Configure pacemaker corosync drbd

‒ Setup HA webserver

37

Preparation• Install openSUSE Leap 42.1

• Configure all repository

• Install all the required software

• Create at least 2 virtul machine with QEMU/KVM

• Configure the Cluster

• Create DRBD

• Activate web server / nginx or apache

• Test the status

Thank you.

Join the conversation,contribute & have a lot of fun!www.opensuse.org

39

Have a Lot of Fun, and Join Us At:www.opensuse.org