JR RichardsonEngineering for the Masses

Hubguru@gmail.com

Virtualizing Asterisk with OpenVZ

Presentation Overview

¬ Discuss various Virtualization technologies¬ Examples for Virtualization with Asterisk¬ Step by step tutorial, build a Virtual Server

for Asterisk¬ Discuss operation and maintenance¬ Considerations, what to watch out for¬ Wrap up

Overview

¬ Basic Asterisk system require very little hardware resources

¬ Many Asterisk embedded PC devices being used

¬ Modern server are overkill for many Asterisk systems

¬ Where need arises, many virtual Asterisk PBX’s can be deployed on 1 physical server

¬ Virtualizing Asterisk improves hardware efficiency

¬ Get more bang for your buck

Virtualization Introduction

Virtualization is a framework or methodology of dividing the resources of a computer into multiple execution environments.

Virtualization techniques create multiple isolated partitions — Virtual Machines (VM) or Virtual Environments (VEs) — on a single physical server

Virtualization Technologies

¬ Virtual Machines (VMs)– Emulate real or fictional hardware– Runs in guest operating system– Greater host CPU load, requires virtual

machine monitor (VMM)– VMware, QEMU, Parallels and MS Virtual Server

¬ Paravirtualization– Also uses VMM– Guest OS is modified to run under the

hypervisor (VMM)– Allows for various Guest OS– Xen and UML

Virtualization Technologies cont

¬ OS Level Virtualization– Most applications on the host can share with

other virtual environments (VEs)– Does not require each virtual machine to run

dedicated OS– Host server runs multiple instances of single

OS, can be different versions– Designed to isolate and secure each virtual

instance to run multiple applications– Very low overhead– OpenVZ, Linux-VServer, Solaris Zones,

FreeBSD Jails

Comparison

¬ VMs have a wide scope of usage, but poor performance

¬ Para-VMs have better performance, tighter integration with Host OS, but have to modify Guest OS

¬ OS level provides best performance, simpler to administer, can natively access all VEs from host, best choice for same OS server consolidation

¬ Personal Choice Open Source Project OpenVZ– Very Mature, has kernel for many Linux

distributions, great community and support– http://wiki.openvz.org

Asterisk Consideration

¬ Base OS mostly Linux¬ Same Linux Distribution used¬ Same kernel used¬ Common Asterisk versions and packages¬ Once version is selected, consecutive PBX

builds will be same¬ This is optimal for OS level Virtualization¬ Can take advantage of scalability, ease of

setup and management

Why Virtualize Asterisk?

¬ Service Providers– Offer hosted PBX solutions– Individual PBX for each customer– Simpler than Clustering Asterisk servers

¬ 2 Asterisk PBX’s on 1 Server– Soft switch application redundancy

¬ Limited hardware resources– Run production PBX on same hardware while testing

other versions or custom applications

¬ Lab– Testing various versions of Asterisk– Mock up production systems– Build Cluster arrangements for testing

Build a Virtual Server with OpenVZ

¬ Hardware Considerations– Fast processor, multi core, multi proc– Lots of RAM– Large and fast disk drive, SATA, SCSI

¬ Choose base Linux Distro– OpenVZ has several kernels for pre-packaged

linux distributions, Debian, Fedora, CentOS, SUSE, Slackware, Gentoo, Ubuntu

– Can also compile kernel package directly¬ For the following example we will use

Debian Etch with pre-build OpenVZ kernel

Install Base OS and OpenVZ Kernel

¬ Various Linux Distro install instructons on wiki– http://wiki.openvz.org/Category:Installation

¬ Example Hardware Node Specs– Asus P5MT Mobo– 3.2Ghz P-4 Proc– 4 Gig RAM– 80 Gig SATA 7200 Hard Drive

¬ BIOS setting– Disable serial and parallel prorts– Power recovery set to ‘last state’– Wait for F1 errors ‘disable’ for no keyboard

Install Base OS and OpenVZ Kernel

¬ Debian Base system, typical install– 3 partitions, / and /var and swap, majority of

space for /var– Do not install any optional packages– Virtual containers reside in /var/lib/vz/private/

o For disk quota to perform as expected, container space must not impede the root partition

– Delete CDROM as source in /etc/apt/source.list– apt-get update– apt-get dist-upgrade– Reboot so kernel updates– apt-get install ssh ftp

Install Base OS and OpenVZ Kernel

¬ Add OpenVZ repository to /ext/apt/source.list– deb http://download.openvz.org/debian etch main– deb-src http://download.openvz.org/debian etch main– deb http://debian.systs.org/ etch openvz– deb-src http://debian.systs.org/ etch openvz

¬ wget http://download.openvz.org/debian-systs/dso_archiv_signing_key.asc

¬ apt-key add dso_archiv_signing_key.asc¬ wget

http://debian.systs.org/dso_archiv_signing_key.asc¬ apt-key add dso_archiv_signing_key.asc¬ apt-get update

Install Base OS and OpenVZ Kernel

¬ apt-get install linux-image-2.6.18-openvz-13-1etch5-686 linux-headers-2.6.18-openvz-13-1etch5-686

¬ Reboot into OpenVZ kernel, verify ‘uname –r’¬ apt-get install libncurses5-dev bison libssl-dev

libnewt-dev zlib1g-dev procps gcc make binutils modconf g++ debootstrap ntp

¬ cd /usr/src/ get zaptel and install– ‘mknod /dev/zap/transcode c 196 253’– ‘chmod 660 /dev/zap/transcode’

¬ Check for rtc irq conflict, 1000Hz lost sync error– Edit grub menu.lst, add acpi=off to kernel switch

Install Base OS and OpenVZ Kernel

¬ apt-get install vzctl vzquota vzdump¬ ‘cd /’

– ‘ln –s /var/lib/vz vz

¬ Install template, ‘apt-get install [template]’– vzctl-ostmpl-debian - OpenVZ - OS Template -

Debian 4.0– vzctl-ostmpl-debian-4.0-i386-minimal - OpenVZ

- OS Template debian-4.0-i386-minimal– vzctl-ostmpl-debian-5.0-i386-minimal - OpenVZ

- OS Template debian-5.0-i386-minimal

Operation and Maintenance

¬ Config files in /etc/vz/¬ OpenVZ tools

– vzctl– vzquota– vzdump– vzlist– vzmigrate– vzcalc– vzmemcheck– vzsplit– vzcfgvalidate– beancounters

VE Config File /etc/vz/conf/101.conf

ONBOOT="yes"# UBC parameters (in form of barrier:limit)# Primary parametersNUMPROC="200:200"NUMTCPSOCK="300:300"# Secondary parametersKMEMSIZE="4874532:5058032"TCPSNDBUF="1054834:2093234"TCPRCVBUF="1054834:2093234"SHMPAGES="28842:28842"NUMFILE="3000:3000"# Disk quota parameters (in form of softlimit:hardlimit)DISKSPACE="1548576:1653434"# CPU fair sheduler parameterCPUUNITS="7000"VE_PRIVATE="/var/lib/vz/private/$VEID"OSTEMPLATE="ast4-1-2-24-ovz"NAMESERVER=“10.10.14.1"IP_ADDRESS=“10.10.14.101“DEVNODES="tty9:rw zap/channel:rw zap/ctl:rw zap/pseudo:rw zap/timer:rw

zap/transcode:rw"HOSTNAME="pulsion-ve501"

vzsplit

¬ ‘vzslpit’¬ Enter the number of VEs: 20The optimal swap space size is 7088 Mb, twice bigger than the RAM size

# Configuration file generated by vzsplit for 20 VEs# on HN with total amount of physical mem 3544 Mb# low memory 856 Mb, swap size 7632 Mb, Max treads 8000# Resourse commit level 0:# Free resource distribution. Any parameters may be increased# Primary parametersNUMPROC="400:400"NUMTCPSOCK="400:400"VMGUARPAGES="138035:2147483647"# Secondary parametersKMEMSIZE="8979701:9877671"TCPSNDBUF="1354834:2993234"TCPRCVBUF="1354834:2993234"SHMPAGES="54437:54437"NUMFILE="3488:3488"DISKSPACE="268134:294948"DISKINODES="164219:180642"CPUUNITS="7148"

Routine Operation and Maintenance

¬ Create VE¬ Create and modify templates¬ Check beancounters¬ List running VEs¬ Migrate VE to other hardware nodes¬ Monitor

– Log files– Disk space– Proc utilization– Main memory

Working with Templates

¬ ‘vzctl create 101 --ostemplate debian-4.0-i386-minimal’– OK to do this in production, Kernel guarantees VE

resources¬ Edit /etc/vz/conf/101.conf

– NAMESERVER=“10.10.14.1"– IP_ADDRESS="10.10.14.101"– DEVNODES="zap/channel:rw zap/ctl:rw zap/pseudo:rw

zap/timer:rw zap/transcode:rw“¬ ‘vzctl start 101’¬ ‘vzctl enter 101’¬ Update /etc/apt/source.lst with OpenVZ repositories¬ ‘apt-get update’¬ ‘apt-get upgrade’¬ ‘apt-get install linux-headers-`uname –r`’

Install Asterisk in Template

¬ apt-get install libncurses5-dev bison libssl-dev libnewt-dev zlib1g-dev procps gcc make binutils ftp g++ ntp ssh ftp

¬ Download and install asterisk, zaptel not needed in VE, has access to hardware node zaptel through devicenodes

¬ Configure generic Asterisk templates¬ Configure ntp, cron jobs, linux house

keeping, smarthost, syslog, etc …– Build this as the template PBX system

Update the Template

¬ Stop VE and take a snapshot– ‘vzctl stop 101’

¬ Go into the VE directory– ‘cd /vz/private/101’

¬ ‘tar -zcf /vz/template/cache/ast4-1-2-24-ovz.tar.gz .’– don’t forget the end dot

¬ Update /etc/vz/dist and /etc/vz/conf files to match the name of the new template

¬ Distribute new template files to other hardware nodes

vzctl

¬ Create and Destroy VE– ‘vzctl create 101 --ostemplate ast5-1-2-24-ovz

--config ast5-1-2-24’– ‘vzctl destroy 101’ removes private area

¬ Start, Stop, Restart VE– ‘vzctl start 101’– ‘vzctl stop 101’– ‘vzctl restart 101’

¬ Status and Enter VE– ‘vzctl status 101’

VEID 100 exist mounted running– ‘vzctl enter 101’

root@ve100:/#

vzlist

¬ ‘vzlist –a’ VEID NPROC STATUS IP_ADDR HOSTNAME 100 18 running - ve100 501 20 running 10.1.1.12 pulsionmedical-ve501 502 20 running 10.1.1.13 veritas-ve502 503 18 running 10.1.1.14 fta-ve503 504 20 running 10.1.1.15 nathealth-ve504 505 20 running 10.1.1.16 ampow-ve505 506 21 running 10.1.1.17 systemusa-ve506 507 32 running 10.1.1.18 frank-ve507 508 21 running 10.1.1.19 gclub-ve508 509 19 running 10.1.1.10 symphony-ve509

beancounters

¬ ‘cat /proc/user_beancounters’ uid resource held maxheld barrier limit failcnt 509: kmemsize 664371 1252967 4874532 5058032

0 lockedpages 0 0 338 338 0 privvmpages 4172 11690 49152 53575 0 shmpages 0 656 28842 28842 0 numproc 19 103 100 200

47 physpages 2492 3710 0 483647 0 oomguarpages 2492 3710 6144 483647

0 numtcpsock 3 4 300 300 0 tcpsndbuf 26784 49104 10834 3234 0 tcprcvbuf 49152 0 14834 93234 0 othersockbuf 24552 105272 577417 45817

0 dgramrcvbuf 0 4648 592096 92096 0 numfile 501 726 3000 3000 0

beancounters perl script

¬ astvz5:~# ./beans2 #################################################################BEANS FOR UID 509resource held maxheld barrier limit failcntkmemsize 639.07 kb 1.19 mb 4.65 mb 4.82 mb 0lockedpages 0 0 1.32 mb 1.32 mb 0privvmpages 16.30 mb 45.66 mb 192.00 mb 209.28 mb 0shmpages 0 2.56 mb 112.66 mb 112.66 mb 0numproc 19 31 200 200 0physpages 9.73 mb 14.49 mb 0 MAX_ULONG 0vmguarpages 0 0 24.00 mb MAX_ULONG 0oomguarpages 9.73 mb 14.49 mb 24.00 mb MAX_ULONG 0numtcpsock 3 4 300 300 0tcpsndbuf 26.16 kb 47.95 kb 1.01 mb 2.00 mb 0tcprcvbuf 48.00 kb 0.00 kb 1.01 mb 2.00 mb 0othersockbuf 23.98 kb 752.80 kb 563.88 kb 1.28 mb 3259dgramrcvbuf 0.00 kb 4.54 kb 578.22 kb 578.22 kb 0numothersock 14 29 300 300 0numfile 501 726 3000 3000 0

vzcalc

¬ ‘vzcalc –v 101’Resource Current(%) Promised(%) Max(%)Low Mem 0.21 3.11 3.11Total RAM 0.30 n/a n/a Mem + Swap 0.10 0.31 n/aAlloc. Mem 0.15 0.31 1.97Num. Proc 0.03 n/a 0.31--------------------------------------------Memory 0.30 3.11 3.11

vzmigrate

¬ Setup ssh root access between hardware nodes– http://wiki.openvz.org/Migration_from_one_HN_to_another

¬ ‘apt-get install rsync’¬ Correct syntaxt in vzmigrate script

– nano +382 /usr/sbin/vzmigrate– -if [ $? != 20 && $? != 21 && $? != 0 ]; then– +if [ $? != 20 ] && [ $? != 21 ] && [ $? != 0 ]; then

¬ vzmigrate -r no --keep-dst –v [ip address of new HN] [VE number to transfer]

¬ vzmigrate -r no --keep-dst -v 10.10.10.2 101¬ Migration ~120 seconds, VE only down 10

seconds

OpenVZ Considerations

¬ Hardware node VE execution scripts– Can use to run commands inside VE from HN

¬ Example of running ‘date’ and ‘ntpq –p’ commandastvz5:~# more vz-exec-date echo "################# VE 501

###################"vzctl exec 501 datesleep 1vzctl exec 501 ntpq -psleep 1echoechoecho "################# VE 502

###################"vzctl exec 502 datesleep 1vzctl exec 502 ntpq -psleep 1echoecho

OpenVZ Considerations

¬ astvz5:~# more vzcalc-all echo "################# VE 501

###################"vzcalc -v 501sleep 1echoechoecho "################# VE 502

###################"vzcalc -v 502sleep 1echoecho

OpenVZ Considerations

¬ astvz5:~# more migrate-all vzmigrate -r no --keep-dst -v 10.10.10.2 501sleep 2vzmigrate -r no --keep-dst -v 10.10.10.2 502sleep 2vzmigrate -r no --keep-dst -v 10.10.10.2 503sleep 2vzmigrate -r no --keep-dst -v 10.10.10.2 504sleep 2

Kernel Issue Workaround

¬ OpenVZ requires at least 1 active VE running to operate as expected

¬ If you have 20 production VEs, run 21 VEs on hardware node

¬ This may have been fixed in later OpenVZ kernels

¬ Beancounters require VE to be stopped at least 5 minutes to clear counters

¬ If VE kmemsize beancounters are hung, need to reboot hardware node to clear

Wrap Up

¬ Open VZ, like most open source projects have quirks but for the most part works straight out of the box

¬ The community is very knowledgeable and helpful. User list users@openvz.org– https://openvz.org/mailman/listinfo/users

¬ 20+ lightly loaded Asterisk VEs can be successfully deployed in production on a single hardware node on moderate to high end servers

¬ Great wiki, everything OpenVZ– http://wiki.openvz.org