building elastix-2.4 high availability clusters with drbd ... · pdf filebuilding elastix-2.4...

1

Building Elastix-2.4 High Availability Clusters withDRBD and Heartbeat (using a single NIC)

Credits A great deal of credit goes out to Daniel Guevara and Amjad Jabali,who authored previous versions of this document. Daniel Guevara'sdocument is linked above, but it appears Amjad Jabali's is offline.

While I have added a great deal to this document, and made manychanges, a great deal of work was done by these other authors, tothe point where this document would not exist without them.

Thanks for the great work guys.

This information has been modified and updated by Nick Ross.Please refer to the original document found at:

Changes made to this document will be explained at the very endin Appendix A.

Document Last updated October 15th, 2015.

2

INDEX

Operational Overview………………………………………………… …What Is DRBD………………………………………………… ….

eat does………………………………………… ………………………………………………………….4

………………………………………….………………………………………… …….…

Credits…………………………………………………………………..……………………………………………………………..

3

Operational Overview

What is DRBD?

4

failover failbackswitchover.

Equipment Overview

yum –y update

6

NOTE

Press “t” to change the partition system ID Press “3” to choose partition number

Press “w” to save changes

RESTART SERVER

Note:

192.168.1.243 voipserver.drbd192.168.1.242 voipbackup.drbd

global { usage-count no; }resource r0 {protocol C;startup { wfc-timeout 10; degr-wfc-timeout 30; } #change timers to your needdisk { on-io-error detach; } # or panic, ...net {

mke2fs -j /dev/sda3

dd if=/dev/zero bs=1M count=500 of=/dev/sda3; sync

yum install heartbeat drbd83 kmod-drbd83

7

Note:

Reference:

after-sb-0pri discard-least-changes;after-sb-1pri discard-secondary;after-sb-2pri call-pri-lost-after-sb;cram-hmac-alg "sha1";shared-secret "Cent0Sru!3z";}syncer { rate 5M; }on voipserver.drbd {device /dev/drbd0;disk /dev/sda3;address 192.168.1.242:7788;meta-disk internal;}on voipbackup.drbd {device /dev/drbd0;disk /dev/sda3;address 192.168.1.243:7788;meta-disk internal;}}

after-sb-0pri discard-least-changes;after-sb-1pri discard-secondary;after-sb-2pri call-pri-lost-after-sb;

scp /etc/drbd.conf root@voipbackup.drbd:/etc/

drbdadm create-md r0

service drbd start

cat /proc/drbd

’

cd /replica

amportal chown

tar -zcvf etc-asterisk.tgz /etc/asterisk

tar -zxvf etc-asterisk.tgz

tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk

tar -zxvf var-lib-asterisk.tgz

tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk/

tar -zcvf var-www.tgz /var/www/

tar -zxvf usr-lib-asterisk.tgz

tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk/

tar -zxvf var-spool-asterisk.tgz

tar -zcvf var-lib-mysql.tgz /var/lib/mysql/

tar -zxvf var-lib-mysql.tgz

tar -zcvf var-log-asterisk.tgz /var/log/asterisk/

tar -zxvf var-log-asterisk.tgz

tar -zxvf var-www.tgz

rm -rf /etc/asterisk

rm -rf /var/lib/asterisk

rm -rf /usr/lib/asterisk/

rm -rf /var/spool/asterisk

rm -rf /var/www

drbdadm -- --overwrite-data-of-peer primary r0

watch -n 1 cat /proc/drbd

mkfs.ext3 /dev/drbd0

mkdir /replica

mount /dev/drbd0 /replica

drbdadm role r0

Note:

Execute ‘df –h’ on the primary to confirm that our /dev/drbd0 partition is

Note:not display the /dev/drbd0 partition unless it’s assuming primary mode.

rm -rf /var/lib/mysql/

rm -rf /var/log/asterisk/

ln -s /replica/etc/asterisk/ /etc/asterisk

ln -s /replica/var/lib/asterisk/ /var/lib/asterisk

ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk

ln -s /replica/var/spool/asterisk/ /var/spool/asterisk

ln -s /replica/var/lib/mysql/ /var/lib/mysql

ln -s /replica/var/log/asterisk/ /var/log/asterisk

ln -s /replica/var/www /var/www

cd /

service mysqld restart

service mysqld stop

service asterisk stop

service httpd stop

service elastix-updaterd stop

service elastix-portknock stop

rm -rf /etc/asterisk

rm -rf /var/lib/asterisk

rm -rf /usr/lib/asterisk/

umount /replica ; drbdadm secondary r0

mkdir /replica ; drbdadm primary r0 ; mount /dev/drbd0 /replica

ls /replica/

drbdadm role r0

10

Heartbeat Configuration

rm -rf /var/spool/asterisk

rm -rf /var/lib/mysql/

rm -rf /var/log/asterisk/

rm -rf /var/www

ln -s /replica/etc/asterisk/ /etc/asterisk

ln -s /replica/var/lib/asterisk/ /var/lib/asterisk

ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk

ln -s /replica/var/spool/asterisk/ /var/spool/asterisk

ln -s /replica/var/lib/mysql/ /var/lib/mysql

ln -s /replica/var/log/asterisk/ /var/log/asterisk

ln -s /replica/var/www /var/www

service mysqld restart

service mysqld stop

service asterisk stop

service httpd stop

service elastix-updaterd stop

service elastix-portknock stop

chkconfig drbd on

chkconfig asterisk off

chkconfig mysqld off

chkconfig httpd off

chkconfig elastix-updaterd off

chkconfig elastix-portknock off

service mysqld stop

service asterisk stop

service httpd stop

service elastix-portknock stop

service elastix-updaterd stop

debugfile /var/log/ha-debug

logfile /var/log/ha-log

umount /replica/ ; drbdadm secondary r0

drbdadm primary r0 ; mount /dev/drbd0 /replica

11

[root@voipserver.drbd ha.d]#

[root@svoipbackup.drbd ha.d]#

Execute ‘ –h’ on the primary to confirm that our /dev/drbd0 partition is

NOTE: I've set auto_failback to off. This seems more appropriate to me.

use the following command on the current secondary to switch back:

sh /usr/lib/heartbeat/hb_takeover

logfacility local0

keepalive 2

deadtime 30

warntime 10

initdead 120

udpport 694

bcast eth0

auto_failback off

node voipserver.drbd

node voipbackup.drbd

chkconfig --add heartbeat

chkconfig heartbeat on

voipserver.drbd drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::192.168.1.245/24/eth0/192.168.1.255 mysqld asterisk httpd elastix-updaterd elastix-portknockvoipserver.drbd MailTo::your@emailgoeshere.com,your@emailgoeshere.com::DRBD/HA-ALERT

auth 1

1 sha1 MySecret

chmod 600 /etc/ha.d/authkeys

service heartbeat start

drbdadm role r0

it doesn’t lose connectivity. Make

Special Note:

Troubleshooting:

Credits

References

Author:

12

tcpdump –i eth0:0 –s 1500 –w captura.pcap #capture traffic

mv captura.pcap /var/www/html #move file to web for download

1

1

1

1

cd /replica

tar -zcvf etc-asterisk.tgz /etc/asterisk

tar -zxvf etc-asterisk.tgz

tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk

tar -zxvf var-lib-asterisk.tgz

tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk/

tar -zcvf var-www.tgz /var/www/

tar -zxvf usr-lib-asterisk.tgz

tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk/

tar -zxvf var-spool-asterisk.tgz

tar -zcvf var-lib-mysql.tgz /var/lib/mysql/

tar -zxvf var-lib-mysql.tgz

tar -zcvf var-log-asterisk.tgz /var/log/asterisk/

tar -zxvf var-log-asterisk.tgz

tar -zxvf var-www.tgz

rm -rf /etc/asterisk

rm -rf /var/lib/asterisk

rm -rf /usr/lib/asterisk/

rm -rf /var/spool/asterisk

rm -rf /var/lib/mysql/

rm -rf /var/log/asterisk/

rm -rf /var/www

ln -s /replica/etc/asterisk/ /etc/asterisk

ln -s /replica/var/lib/asterisk/ /var/lib/asterisk

ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk

ln -s /replica/var/spool/asterisk/ /var/spool/asterisk

ln -s /replica/var/lib/mysql/ /var/lib/mysql

ln -s /replica/var/log/asterisk/ /var/log/asterisk

ln -s /replica/var/www /var/www

cd /

1

2

2

APPENDIX IIP Sourcing Part 2

The previous section ensures that external traffic will be sent from the box using the cluster IP address. What it does notdo, is use the cluster IP address on the internal LAN. This could be a problem for certain equipment on your LAN. For devices thatregister with your asterisk PBX, the line "bindaddr=192.168.1.245" in sip_general_custom.conf will take care of the issue.HOWEVER, a problem still exists with devices that your PBX registers with. For instance, VoipServer.drbd will try registering itselfto another device on the LAN using the IP address 192.168.1.242.-

The only solution to this problem is to specify an IP source address when trying to reach individual hosts on the network.This is not often an issue, but nevertheless is something that you may run into. To fix this, we need to implement a new serviceon our linux system. These steps must be implemented on both the primary and secondary servers.-Step 1- Type the following command:nano /etc/init.d/pbxiprouting-Step 2- Paste the code found on the following page into the editor. YOU MUST CHANGE THE IP ADDRESSES IN THE SCRIPTThere are two entries. One is under start(), the other is under stop(). I've used 192.168.1.29 as an arbitrary IP address. The IPaddress that you use here should represent another system on the internal network that your asterisk PBX will INITIATEcommunication with. A good example would be an Analog Gateway Device, where your server reaches out to it in order to register.It can really be any device on the local network, aside from the servers in our drbd cluster. If you wish to do this for multiple devices,you can copy and paste, entering multiple lines with different IP addresses. Use CTRL+ O and CTRL + X to save & exit.-Step 3- Enter the following command:chmod 755 /etc/init.d/pbxiprouting-Step 4- Verify that the script works, with the commands:-service pbxiprouting startservice pbxiprouting stop-Step 5- If the above works normally, the last step is to add an entry within your /etc/ha.d/haresources file. Change:(....)IPaddr::192.168.1.245/24/eth0/192.168.1.255 mysqld asterisk httpd(...)to(....)IPaddr::192.168.1.245/24/eth0/192.168.1.255 pbxiprouting mysqld asterisk httpd(...)-This change ensures that the necessary routing changes are only made when the cluster is owned by THAT host. It also ensuresthat the routing changes are removed when the host releases the cluster.

#!/bin/bash

# description: pbxiprouting

# process name: pbxiprouting

# Author: Nick Ross

. /etc/init.d/functions

RETVAL=0

getpid() {

pid=`ps -eo pid,comm | grep "asterisk" | awk '{ print $1 }'`

}

start() {

echo -n $"Starting PBXIPRouting: "

route add -host 192.168.1.29 dev eth0:0

RETVAL=0

if [ $RETVAL -eq 0 ]; then

touch /var/lock/subsys/pbxiprouting

echo_success

else

echo_failure

fi

echo

return $RETVAL

}

stop() {

echo -n $"Stopping PBXIPRouting: "

route delete -host 192.168.1.29

RETVAL=0

rm -f /var/lock/subsys/pbxiprouting

echo_success

return $RETVAL

}

# See how we were called.

case "$1" in

start)

start

;;

stop)

stop

;;

status)

getpid

if [ -n "$pid" ]; then

echo "PBXIPRouting (pid $pid) is running..."

else

RETVAL=1

echo "PBXIPRouting is stopped"

fi

;;

restart)

stop

start

;;

*)

echo $"Usage: $0 {start|stop|status|restart}"

exit 1

;;

esac

exit $RETVAL

Script for /etc/init.d/pbxiprouting

APPENDIX JIPSec for DRBD

If you are not using a two NIC configuration, with a secured and separate network for DRBD, its very likely that yourDRBD data is vulnerable while in transit. DRBD transmits raw disk data, without any encryption. Changes to your configuration,passwords, etc., are all transmitted over the wire and vulnerable to interception. Luckily, this is very easy to secure in a linuxenvironment, via IPSec. This will have to be done on BOTH the primary and secondary server.-Step 1- Install the ipsec-tools package. Use the following command:yum install ipsec-tools-Step 2- Make a file to start the ipsec connection. Use the command:nano /etc/sysconfig/network-scripts/ifcfg-ipsec0-Step 3- Enter the following in the test editor (this assumes you are on VoipMain.drbd):DST=192.168.1.243TYPE=IPSECONBOOT=yesIKE_METHOD=PSK-(note: on voipserver, you would change the DST field to DST=192.168.1.242)The DST field always contains the ip of the REMOTE server, NOT the ip of the server you are on.CTRL+O saves, CTRL+X exits the editor.-Step 4- Make a key file. Type the command:nano /etc/sysconfig/network-scripts/keys-ipsec0-Step 5- Choose a key for the ipsec connection (change it from what I put below): Type in something like this in the editor:IKE_PSK=supersecretpassword12345!-CTRL+O saves, CTRL+X exits-Step 6- Secure the file by typing the following command:chmod 600 /etc/sysconfig/network-scripts/keys-ipsec0-Step 7- Repeat this on the secondary server. Please remember to enter the proper IP address on the secondary, and do not simplycopy and paste the same IP address. See step 3 again for clarification of the DST field.-Step 8- To get the tunnel working without a reboot, you'll have to start it manually. On both servers, type the command:ifup ipsec0-That's it, you are done. The ipsec connection should come online automatically when you reboot.-If you'd like to verify the ipsec connection is working, you can use tcpdump like so:tcpdump -n host 192.168.1.242 and host 192.168.1.243-Tcpdump should should an AH and ESP field, indicating the header and payload are protected by ipsec. It may take up to ten secondsbefore you see results.-If you ever want to turn of ipsec, the "ifdown ipsec0" command should be executed on both hosts. To prevent IPSec from startingautomatically upon boot, go back to step 3 and set ONBOOT=no .