AdminCamp

Gelsenkirchen,September 2010

Daniel Nashed

Domino on Linux

About the presenter● Daniel Nashed

– Nash!Com - IBM/Lotus Advanced Business Partner/ISV

– Member of The Penumbra group● an international consortium of selected Business Partners pooling

their talent and resources

– focused on Cross-Platform C-API, Domino® Infrastructure, Administration, Integration and Troubleshooting

● Platform Focus: W32, Win64, Linux, AIX® and Solaris®

– Regular speaker at International Conferences

– DNUG Entusiast, Domino Infrastruktur

– Author of Domino Start Script for Linux/Unix● nsh@nashcom.de, http://www.nashcom.de

● Blog http://blog.nashcom.de

Agenda● Introduction

● Right distribution and Linux version

● Installation

● Performance Tuning

● Troubleshooting

● Q & A -- ask questions any time

Why Notes/Domino on Linux?● More and more customers move their Domino environment

to Linux

– Most time combined with a Company-wide Corporate Strategy ● Only makes sense if Linux is one of your stategic platforms● Domino is a good platform to start with Linux

– Fast growing platform● Server Consolidation

– Multiple Partitions on a single Linux box

– On Windows you should only use one or need Virtualization● Cost reduction when migrating from other platforms

– e.g. AIX, Solaris, iSeries, zSeries● Robust, Fast, Scalable Platform

Why Notes/Domino on Linux?● No Registry

– You know exactly what happens● Better Security

● Better Manageability

● Most cost effective

– Less licence costs● Young talented employees have

Linux skills

● Most vendors have a Linux strategy

● Be smart work free :-)

Domino 32bit on a 64Bit Operating System● Total Memory per Process is 32Bit = 4 GB

● Router / HTTP uses most local process memory

● NSF Buffer Pool is the biggest Shared Memory block (512 MB)

64Bit OS Domino Memory / File-System Cache, .... A lot of room

Domino 32bit / Shared Memory 3GB

Servertask 32bit 1GB

Domino 32bit / Shared Memory 3GB

Servertask 32bit 1GB

Servertask 32bit 1GB

Servertask 32bit 1GB

Servertask 32bit 1GB

Servertask 32bit 1GB

4GB Total

Memory Limits● Domino 6

– 2 GB shared memory / 2 GB local process memory per process● Domino 7/8 on SLES/RHEL 32bit

– 3 GB shared memory / 1 GB local process memory per process● Domino 7/8 on SLES/RHEL 64Bit

– More than 3 GB of shared memory

– Only limit is 4 GB total memory size per process ● Domino 7/8 on SLES/RHEL in 64Bit Mode supports unlimited

number of partitions with 32Bit dedicated address space per Domino Partition (DPAR)

– Practical limit would be 3-4 partitions with 12-16 GB of RAM

– you need to take care to have fast disk subsystems

Linux/Unix Partitioning● Multi-Tasking/Multi-User environment

– Domino partition runs with standard user account per partition● Very effective process scheduler and memory manager

– File-System cache uses all remaining memory

– No idle resources● Strong Linux 64-Bit implementation

– Domino runs in 32-bit dedicated memory area per partition

– You can run different Domino Versions on multiple partitions● Scalable file-systems

– Less fragmentation than on NTFS● Mountable file-systems (easier separation of multiple file-

systems)

Hardware Example HP DL380 G5● Affordable mid size hardware – 2U size with local Disks

– Up to 2 Intel Xeon 5X00 sequence Dual-Core or Quad-Core

– Up to 64GB of 667MHz DDR2 Fully-buffered DIMMs

– 4 available PCI-Express expansion slots● RAID Controllers

– HP Smart Array P400 512MB with BBWC (Performance models)

– Expandable storage for support of up to eight (8) SAS or SATA small form factor drives

Customer Linux Example Two Locations - 6 Boxes● Spread multiple Domino Partitions (DPARs)

– On multiple physical machines

– Each machine has one counter part on a different phyical box

● 3 Partitions each

– Balance two busy and

– one lower profile DPAR● Leverage Domino Clustering

– No SAN mirroring!

Server1

Location A Location B

Fail-overLoad balancing

Fail-overLoad balancing

ServerE1

ServerE2

ServerE3

ServerD4

ServerD5

ServerD6

M1a

G1a

A1a

M2a

M3a

Admin

M4a

T1

A2a

M1b

M2a

T1a

G1b

M3b

A2b

A1b

M4b

LDAP

SAN A SAN B▬ 2 Quad-Core CPUs▬ 16 GB RAM▬ SAN disk for data▬ Local Disks for TL

Linux Distributions?● SuSE, RedHat Enterprise are the only supported distributions

– Different Releases of Domino support different Versions of each distribution

– Take care that you use the right distribution else your Domino server might not run and is totally unsupported

● Different versions use different libraries, stack sizes, Java runtime, ...

– Using LD_ASSUME_KERNEL=xyz is not really a solution

Linux Compiler Background● Domino 7 is build with

– GNU Compiler Collection (gcc) version 3.2.2● Domino 8.x is build with

– GNU Compiler Collection (gcc) version 4.1.0● That's why Domino 8 does not run on earlier Linux versions

● If you run Domino 7 on SLES 10 / RHEL 5 you have to install the compat libs

– See next slides

– You also need this if 3rd party applications have not been recompiled!

– But it works and is supported when you install the compat libs

LD_ASSUME_KERNEL???● OS environment variable for overriding the glibc run time

loaded by a particular application

– LD_ASSUME_KERNEL=2.4.20 ● NPTL (default mode for RHEL 3 and SLES9)

– LD_ASSUME_KERNEL=2.4.19 ● Standard LinuxThreads with floating stacks

– LD_ASSUME_KERNEL=2.2.5 ● LinuxThreads with fixed stack size.

● Check glibc version via

– getconf GNU_LIBPTHREAD_VERSION● returns e.g. NPTL 0.61 or linuxthreads-0.10

Recommended Combinations● Domino 8.x and 8.5.x is only supported on SLES10 /RHEL5!

– My preference: SLES10 SP3 / SLES 11

– But RHEL also works great● Domino 7 does need “compat” libs to run on SLES10

– D7 is compiled with older compiler

– See details on next slides● Notes 8.5 Client

– SLED/RHED 10 with current SP

– My preference: Ubuntu 8.04 LTS

– Ubuntu 8.x and 9.x is unsupported and you can and will run into issues!

– New, Ubuntu 10.04 LTS will be supported with 8.5.2 soon● Works fine but tests to support it official are not done yet

Do's and Don'ts for Domino on VMware● VMware ESX is a good platform for test and demo

environments

● For larger mail and application servers native Linux with partitioned servers is recommended

– Less complexity and overhead

– Better resource scheduling

– Smaller servers that cannot be clustered are still a good candidate for VMware to ensure disaster recovery

● E.g. Admin Server● In case you install productive servers on Linux check

– www.vmware.com/pdf/vmware_timekeeping.pdf

– Very detailed information about timer implementation

Installation Steps● Copy and extract binaries

– Binary ftp (or WinScp) to Linux box● Extract tar file via tar -xvf d852_linux.tar

● Start installation

– switch to root

– cd linux

– ./install● Install Options

– notes user and group

– binary location: /opt/ibm/lotus

– data location: /local/notesdata●

Configuration Steps● Very simple and quite similar to Windows

– Now that Domino uses the same installer and configuration on all platforms

● Remote Configuration

– /opt/ibm/lotus/bin/server -listen [port]● Standard port is 8585, can be changed via port option

– connect via serversetup.exe from Notes Admin Client● Local Configuration

– Launch and do configuration via X11 session● Either local or remote

Unix Filesystem Structurefor Partitioned Servers● /local/notesdata is standard for single partition

● Proposed file-systems per partition - each on different disk !

– /local/notes1/● data● translog● viewrebuild

● Use view_rebuild_dir=/local/notes1/viewrebuild/ (3-4 GB)

● /var for system log files

● Swap file (up to two devices)

– at least the same than physical memory

Remote Setup Partitioned Servers● Domino on Windows stores partition information in Registry

– Domino on Linux/Unix stores partition information in /opt/ibm/lotus/.install.dat

– Note: ls -la is needed to list files starting with a dot

data_directories { "/local/notesdata" { data_UNIX_gid = 101 data_UNIX_group = "notes" data_UNIX_uid = 100 data_UNIX_user = "notes" }}ddir_data { "/local/notesdata" { size = 604309335 }}

Locale Settings on Unix● Locale defines Date, Time, Currency and other settings

– You need to set it up properly to get the right results within your Domino applications

– Set Unix Language● Example: export LANG=de_DE or en_US

● Sync Timezone/DST settings on Unix level and Domino!

– Domino for Unix does not sync DST/TimeZone information from OS in all releases

– Use DSTLAW (example for EMEA: DSTLAW=3,-1,1,10,-1,1)● Check “Show locale” & “Show timezone” via console for

proper setup

Locale Settings in notes.ini● Some locale settings are not read from OS (check notes.ini)

– DateOrder=DMY

– ClockType=24_HOUR

– DateSeparator=.

– TimeSeparator=:

Kernel Parameter Tuning● You need the following line in all /etc/pam.d/* control files

like /etc/pam.d/login,su,... to ensure limits can be set

– session required /lib/security/pam_limits.so

– SLES10 does provide this out of the box● Number of open files for notes

– /etc/security/limits.conf

– notes soft nofile 49152

– notes hard nofile 49152● Number of processes/threads notes

– user/etc/security/limits.conf

– notes soft nproc 12500

– notes hard nproc 12500

New since SLES10● /etc/init.d/su now contains

– session include common-session

● common-session contains

– session required pam_limits.so

– session required pam_unix2.so

● This works fine out of the box for Domino in SLES10

– No modification needed in SLES10

– But this is in conflict with the old SuSE start scripts!

Domino tunekrnl● tunekrnl is invoked as part of the start script

– runs with root privileges (sticky bit) and sets tuning parameters● /proc/sys/fs/file-max has been set to "131072".● /proc/sys/kernel/shmmax has been set to "268435456".● /proc/sys/kernel/sem has been set to "250 256000 32 1024".

● /proc/sys/net/ipv4/tcp_fin_timeout has been set to "15".● /proc/sys/net/ipv4/tcp_max_syn_backlog has been set to "16384".● /proc/sys/net/ipv4/tcp_tw_reuse has been set to "1".● /proc/sys/net/ipv4/ip_local_port_range has been set to "1024

65535".

● /proc/11568/mapped_base has been set to "16777216".– set the mapped base to allow 3 GB shared memory

Additional Performance Tuning● Server_Pool_Tasks=80 (default: 20)

– Maximum number of initial thread-pool tasks (IOCP threads)

– Server_Max_Concurrent_Trans= Server_Pool_Tasks * Domino Ports

– Number of concurrent I/O threads to run (throttle to reduce CPU usage and Context-Switches)

– Always tune both at the same time● NSF_DbCache_MaxEntries

– In case of high statistical value inDbCache.OvercrowdingRejections set it either the maximum number of concurrent users or the maximum number of databases open (whatever is higher)

– e.g. NSF_DbCache_MaxEntries=3000

Special Performance Tuning● IPC/Shared Memory (Domino specific)

– Domino allocates most of the internal memory via shared memory

– Depending on the block size this can cause memory fragmentation and performance issues

– export Notes_SHARED_DPOOLSIZE=8000000 (8MB) to set the size shared memory segments allocated

● Set the "noatime" attribute flag to disable the file access modified timestamp on all filesystems for Domino, /var, /tmp in /etc/fstab

● Some additional parameters to check (experimental)

– /proc/sys/vm/buffermem

– /proc/sys/vm/bdflush

Nagle Algorithm on Unix/Linux● The Nagle Algorithm (John Nagle) is designed to optimize

small packets (like for telnet sessions)

– Small packets are combined to larger packets – when the next packet to client is sent

– In case of Domino this can cause delays because individual transactions might be delayed

– The wait time is up to 200 ms!

– Notes transactions are serialized, no other packets need to be send to client at the same time

– The Nagle Algorithm should be disabled for Domino on all Unix/Linux platforms

– Disabled by default in 8.5, Previous releases, notes.ini setting: debug_pd_nagle_off=1

Nagle On/Off Results● Example: 1000 Note Open Transactions

– With Nagle Algorithm ● Average: 62 ms

– Disabled Nagle Algorithm● Average: 5 ms

● Results in detail / Response time distribution

– Before / After

[ 0] -> 658 [ 15] -> 125 [ 16] -> 192 [ 31] -> 15 [ 32] -> 5 [ 47] -> 3 [ 62] -> 1 [ 78] -> 1

[ 0] -> 547[ 15] -> 55[ 16] -> 102[ 31] -> 23...[ 203] -> 15[ 218] -> 71[ 219] -> 48[ 234] -> 59[ 249] -> 12[ 250] -> 15[ 265] -> 10...

Start/Stop Scripts● Redirect console output stream to a file for problem analysis

– some debug messages are only written to console

– crash info will only written to console● Redirect console input stream from a file to allow local

console

– Example: /opt/lotus/bin/server < console.in >> server.log 2>&1 &

– Attach to input and output files for a kind of "life console"

– More a troubleshooting mode. Better use remote console ● Free cross platform start script

– http://www.nashcom.de/nshweb/pages/startscript.htm

– more than a start script.

– allows to start/stop,monitor & troubleshoot your server (NSD...)

Install Incremental Installers / Hotfixes● Hotfix is only aware of the standard location for binaries and

data

– If you switch chose a different binary directory you need an environment setting to specify the location

– Example: export NUI_NOTESDIR=/opt/domino

– This allows the installer to find .install.dat● .install.dat contains all information about the installed

Domino version

Mixed Casing Problems● Customers migrating from W32 have problems with mixed

case filenames

– Cache issue with lower upper case characters for Notes and Web

– Mailfiles, Desktop, Agents, ....

– There is no out of the box solution yet - still! ● Nash!Com has a (workaround-)solution based on an

Extension-Manager routine

– Free on Linux -- commercial on other platforms

– Translates all requests to lowercase (Open, Create, Delete...)

– Works fine if all database and directory names are converted to lowercase during migration

– Has additional logging to trace problems (prints error codes from API calls)

Linux Basics● Physical console either on screen or serial interface

– Used to show startup information and used in single user mode when no network is available

● Normal users login via terminal on either telnet (port 23) or Secure Shell - SSH (port 22)

– As soon network is started ALL operations can be done from ANY terminal. No need for a physical console

– Many users can work with console access totally separate from each other -- multi-tasking / multi-user operating system

– Each user has his login-shell providing the interface to the system

– You can also login using a graphical X-Window terminal local or remote● You have to install and start X-Windows/xdm (runlevel 5) ● Runlevel 3 provides a text based interface

Boot-Process and Run-Levels● Runlevels are used to provide different level of services

– Diffent services are started depending on runlevel

– Runlevel ● 0 - Halt System● 1 - Single User Mode● 2 - No Network (similar to runlevel 1)● 3 - With Networking, Standard Services and Applications● 4 - Unused● 5 - Same as run-level 3 + X-Window (xdm)● 6 - Reboot

● Domino is started in runlevel 3 or 5

– Servers usually use runlevel 3

– You can switch runlevels via "init <n>"

Typical Linux Commands● Shell commands are similar to DOS command window

– But most commands have different names● ls - lists content of a directory (e.g. ls -l)● cd - switch directory● pwd - prints current working directory● mv - move (rename) file● cp - copy file● mkdir - creates a new directory● rm - removes files (e.g. rm -r mydir)● cat - lists content of a file● more - lists content page by page● tail - shows end of a file (e.g. tail -f lists file and waits for more data)● ln - creates symbolic links (e.g. ln -s tools/startup myservertask

Typical Linux Commands● Search for and in files

– grep - finds text in files

– e.g. grep "my text" *.*

● find - finds files

– e.g. find /local/notesdata -name "*.nsf"

– find . -type d -user notes -print

– find . -type f -and -size +100k -print

Typical Linux Commands● Free space on drive

– df - disk free (e.g. df -k) shows free space in KB

● Used space

– du - disk use (e.g. du .)

● Mounted file-systems

– mount - shows all mounted file-systems

Unix File Permissions● Each file has an owner and group

– You can specify read/write/execute right for

– Owner, Group and World (all other users)

– Example: ● chown -R notes:notes * - changes all files to be owed by group and

user notes● chmod 755 * - changes permissions to RWX for owner● and R-X for group and world

– Multiple ways specifying rights● text: g / o / u / a +-= r / w / x● older numeric-style: 1=x, 2=w, 4=r e.g. 755

Linux Prozesse● top - shows the most busy processes

● ps - shows processes

– e.g. "ps -ef" shows all processes

– "ps" just shows own current processes)

Network Related Commands● ifconfig - Network Configuration

● netstat - Network status

– e.g. netstat -an

– shows all open connections and listening ports● /etc/init.d/network start|stop|restart

– starts/stops/restarts networking

Performance Monitoring & Troubleshooting● Platform Statistics

– Great way to get an overview on system performance

– Comparable stats cross all platforms – but minimum interval is 1 min!● OS level details are described in events4.nsf

● top, nmon

– Information about processes -- nmon was originally developed for AIX

● vmstat

– Virtual memory activity and CPU statistics ● iostat -x

– I/O statistics and activity ● netstat -i

– Summarizes network activity– Needs sysstat package for iostat and sar

vmstat● Provides information about CPU, Disk I/O, Memory, Paging

● Gives indication of performance issues

– For high us/sy columns (user/system CPU) check r (run-queue)

– In case run-queue >> number of CPUs your system is CPU bound

– For high numbers of wa (wait IO in %) check iostat -x to check if machine is I/O bound

– High number of cs (context switches) e.g. more than 5000-8000 context switches per partition you might have a priority inversion

vmstat 1 100procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 100764 40612 201828 0 0 9 4 1002 391 8 13 79 0 4 0 0 100764 40612 201828 0 0 0 0 1046 1247 13 48 40 0

Tools used in this Workshop● VMware Workstation

– Just for convenience but free Vmware server would also work

● A great free Telnet & SSH client: Putty

– http://www.chiark.greenend.org.uk/~sgtatham/putty)

● WinSCP - Windows based copy program using SSH

– http://winscp.sourceforge.net

iostat● Provides information about detailed disk activity

● Gives indication of performance issues

– If %util > 80% check await

– Svctm – time spent servicing the request – only disk activities

– Should be below 10 msec

> iostat -xLinux 2.6.16.46-0.12-default (acme-notes-01) 10/06/2007avg-cpu: %user %nice %system %iowait %steal %idle 0.65 0.02 1.42 0.44 0.00 95.08Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %utilsda 0.26 2.57 0.54 0.84 28.71 29.04 14.35 14.52 41.81 0.08 59.29 4.96 0.69sdb 0.14 7.44 0.38 0.24 31.78 54.92 15.89 27.46 141.12 0.02 26.92 4.82 0.30hdc 0.00 0.00 0.00 0.00 0.01 0.00 0.01 0.00 7.85 0.00 6.53 5.39 0.00

Q&A● Questions?

● Contact

– nsh@nashcom.de

– http://www.nashcom.de

– http://blog.nashcom.de