notes/ domino 9.0.x best practices, high availability and...

SNoUG, 13.11.2014 in ZürichDaniel Nashed, Nash!Com

Notes/ Domino 9.0.xBest Practices, High Availability and Web

About the presenter

● Daniel Nashed

– Nash!Com – IBM Business Partner/ISV

– Member of The Penumbra group

● an international consortium of selected Business Partners pooling their talent and resources

– focused on Cross-Platform C-API, Domino® Infrastructure, Lotus Traveler, Administration, Integration and Troubleshooting

– Platform Focus: Windows, xLinux and AIX®

– Regular speaker at International Conferences

– DNUG Enthusiast IBM Lotus® Domino Infrastructure

– Author of the Domino Start Script for Linux and Unix

● [email protected]

● http://www.nashcom.de

Agenda

● Current Security Discussion

– POODLE, SSL, TLS 1.x, SHA-256

● Performance

– Current Performance “Tips”

● Domino 9 Clustering

– NRPC, HTTP and other protocols

● Browser Plugin

– What is it? What to use it for?

● Q & A - ask questions any time

Current Security Discussion

The POODLE Attack

● The POODLE Attack changed the world of SSL

– Padding Oracle On Downgraded Legacy Encryption

● Attack against SSL 3.0

– In combination with a downgrade attack on servers supporting TLS 1.x and SSL 3.0 this can hit even servers supporting TLS 1.x

● The final solution is to disable SSL 3.0 and below

– But there are still applications not supporting TLS 1.x

● Therefore ensuring that a downgrade attack cannot happen is important

– The TLS_FALLBACK_SCSV protocol functionality ensures that only clients that don't support/request TLS will use a lower version like SSL 3.0

SSL Versions

● SSL 3.0

– defined in RFC 6101, 1996

● TLS 1.0

– defined in RFC 2246, January 1999

– It's the next version after SSL 3.0 (aka SSL 3.1)

● TLS 1.1

– defined in RFC 4346, April 2006

● TLS 1.2

– defined in RFC 5246, August 2008

– Supports SHA-256 and ciphers based on that

References for the “POODLE Attack”

● The Register has a good overview article

– http://www.theregister.co.uk/2014/10/14/google_drops_ssl_30_poodle_vulnerability/

● Official Google Information

– https://www.openssl.org/~bodo/ssl-poodle.pdf

● Very technical article “How POODLE happened”

– https://www.dfranke.us/posts/2014-10-14-how-poodle-happened.html

https://www.openssl.org/~bodo/ssl-poodle.pdf

Why do we need TLS support right now?

● Browser vendors will stop SSL 3.0 support Nov/Dec 2014!

– This means all servers need to support TLS 1.0 or higher

● Most applications do support TLS 1.0 at least

– But not all applications correctly negotiate the right protocol version

– Sometimes older applications need to specify the right protocol version manually

– So even an application supports TLS 1.0 it could happen that the lack of SSL 2.0 handshaking can cause negotiation issues

● Disabling SSL 3.0 completely now is not a good idea

– The right first step is to support at least TLS 1.0 and implement TLS_FALLBACK_SCSV

● But the client also needs to support SCSV

– But some servers already disabled support for SSL 3.0 completely

● This means you need an application with TLS 1.0 support now!

● In case of unpatched Notes Clients internet protocols might not work any more!

– Most Java applications should be safe

Firefox Plans

● https://blog.mozilla.org/security/2014/10/14/the-poodle-attack-and-the-end-of-ssl-3-0/

● “SSLv3 will be disabled by default in Firefox 34, which will be released on Nov 25”

– This means auto updated clients will not be able to open any SSL 3.0 enabled website by default

– Can still be enabled

● “As an additional precaution, Firefox 35 will support a generic TLS downgrade protection mechanism known as SCSV. If this is supported by the server, it prevents attacks that rely on insecure fallback.”

Domino Interims Fix Introduces TLS 1.0

● Available since 4. November 2014

● For all Platforms and supported Versions

– 9.0.1 FP2, 9.0, 8.5.3 FP6, 8.5.2 FP4, 8.5.1 FP5

● First Version is a Server Fix

– Only Standard Client has also shipped because of Cert Request SHA2 changes

– Other Clients will follow soon with an unrelated Java patch

● TLS 1.0 support for all Internet Protocols inbound and outbound

– HTTP, SMTP, LDAP, POP3, IMAP & DIIOP

– Support for TLS_FALLBACK_SCSV

– First version does not allow to disable SSL 3.0 completely

– Cipher suite list for outbound connections re-ordered to place AES ciphers first

–

Details about the Interims Fix

● Removed support

– SSLv2

– SSL renegotiation has been disabled

– All weak (<128 bits) cipher suites have been disabled

● No UI Changes in HTTP Configuration

– The fix will override existing configuration with support for TLS 1.0

● Notes.ini DEBUG_SSL_HANDSHAKE=2

– Will show the protocol version used

● Reference

– http://www.lotus.com/ldd/dominowiki.nsf/dx/IBM_Domino_TLS_1.0

What happens to other applications?

● Applications based on OpenSSL

– Older versions might need to specify TLS 1.0 explicitly

– Newer versions work without any change

● It is strongly recommend to keep security libs like OpenSSL updated anyway!

● Java

– Java 1.6 supports TLS 1.0 (Notes/Domino currently ships with IBM Java 1.6)

– Java 1.7 supports TLS 1.0, 1.1, 1.2

– Most applications should work unchanged

● Current Mobile devices support TLS

– We did not ran into any issues yet – Also Traveler connections work fine even when disabling SSL 3.0 completely

● You need to test all your applications using SSL connections

– Including Secure LDAP Connections for example in Directory Assistance

Secure Proxies

● Many customers use Reverse Proxies and other SSL enabled proxies

● You need to review those servers and check their SSL/TLS support

– This is not only true for inbound connections to the Proxy but also to the internal severs

– Most Secure Proxies do support TLS already

– Many servers (for example Apache based servers) have still SSL 3.0 and older SSL enabled

● Check SSL level supported and also cipher types!

● Good article about disabling SSL 3.0 and weak ciphers on Apache

– https://raymii.org/s/tutorials/Strong_SSL_Security_On_Apache2.html

– In short: Ensure Open SSL is up to date and pass the right parameters to OpenSSL

● SSLProtocol All -SSLv2 -SSLv3

● SSLCipherSuite AES256+EECDH:AES256+EDH

SSL Test Tools

● Probably one of the most busy SSL Test Sites those days

– Can be used to get an idea about your server security status

– Will provide a a “rating” for your server from A to F

– Also includes details about supported SSL protocol version and ciphers

● Also contains a simulation what ciphers certain applications might use

– There is also a test to check which SSL protocol version and ciphers are supported

● Server Test

– https://www.ssllabs.com/ssltest/

● Client Test

– https://www.ssllabs.com/ssltest/viewMyClient.html

● Even Google still supports SSL 3.0 today

SHA-1 is rated as “insecure”

● SHA-1 is not recommended any more

– There are at least theoretical attacks against SHA-1

– Customers are encouraged to move away from SHA-1 to avoid situations we had before with MD5

– SHA-256 is recommended and required for secure encryption

– Governments recommend to move to SHA-256

– SHA-256 is approved by Federal Information Processing Standard (FIPS) 140-2

● Browser vendors decided start to warn when using SHA-1 certificates

– For example: Google starts first to warn for certificates expiring end of this year

● Reducing step by step the expiration time for the certs (1.1.2017, .. 1.1.2016)

– Affected certificates are all Server and intermediate CAs singed with SHA-1

– Root Certifiers are not affected because they are verified in a different way

Browser Vendors start to sunset SHA-1

● This means that you have to replace your certificates ASAP

– Best practice is also to create a new public/private key and not to send a new certificate request (CSR) for an existing key!

● Key could have been compromised and you don't know about it yet

– Ensure that the CA you are using already supports SHA-2

● Most CAs only support SHA-2 today because for exact those reasons

– If you server certificate expires later than 31.12.2015 and your server does not support SHA-2 yet, consider requesting a cert with a shorter valid period

● Just a work-around. Better would be to update your server or put a secure reverse proxy in front of it

● References

– https://blog.mozilla.org/security/2014/09/23/phasing-out-certificates-with-sha-1-based-signature-algorithms/

– http://googleonlinesecurity.blogspot.de/2014/09/gradually-sunsetting-sha-1.html

SHA-256 (SHA-2) Support

● Domino 9.0.x already supports SHA-256 in some areas

– X.509 certificate signature verification and S/MIME signed mail

– some areas of Notes/Domino where a password such as the Internet (HTTP) password was previously "hashed."

● Side Note: You should switch to the version 8 more secure internet password!

– “Password verification compatible with Notes/Domino release 8.01 or greater”● Directory Profile and update existing Person-Docs with Agent

– Internet CA supports SHA-256

● Domino 9.0.1 FP2 IF1 supports SHA-256 Certificates for all Internet Protocols and for Keyring Files

– New Keyring files Management Tool “kyrtool”

New Keyring Tool - “kyrtool”

● Separate Download

– Available for Win32/64, Linux 32/64 on Client or Server

● Can be used to import, show, export certificates

– But not to create a private/public key and a certificate request

● You can use OpenSSL to create the key and the request

– Or you can use any other tool to create the key and the request

– Or use an existing key and cert in PEM format

● Importing Trusted Roots

– Either add it to the PEM file from leave to note (key, cert, intermediates, root)

– Importing roots separately needs Notes/Domino 9.0.1 FP2 IF1 code

● Backend API change is needed

Create a Certificate using OpenSSL

● OpenSSL

– available native installed on Linux/Unix

– On Windows you can use a cygwin environment

● Create a Private/Public Key

– openssl genrsa -out server.key 4096

● Generate a Certificate Signing Request (CSR)

– openssl req -new -sha256 -key server.key -out server.csr

● Send CSR to CA for signing

– Or create a self singed certificate for testing

● openssl x509 -req -days 3650 -sha256 -in server.csr -signkey server.key -out server.pem

– Result is a “PEM” file

Create Keyring File

● Create a new Keyring File

– kyrtool create -k c:\lotus\notes\data\keyring.kyr -p password

– When creating a keyring file you need to specify a password

● All other commands will read the password from the “.sth” file

● Importing Key, Certificate, Intermediates and Trusted root

– Copy key, cert, intermediates and root into one PEM file

– kyrtool import all -k c:\lotus\notes\data\keyring.kyr -i c:\lotus\notes\data\ssl\server.txt

● You can als import the different parts separately

– Kyrtool import all|keys|certs|roots -k c:\keyring.kyr -i c:\server.txt

– But that makes the import a lot more complicated

Keyring “show” command

● Can be used to show information from a keyring file

● Kyrtool show certs -k c:\keyfile.kyr

– Shows all certs including the root machting the cert

● Kyrtool show keys -k c:\keyfile.kyr

– Shows all keys in the keyfile

● Kyrtool show roots -k c:\keyfile.kyr

– Shows all trusted roots in the keyfile

● Verbose option “-v” can be used to dump more detailed information

Verify Import File

● Before importing a PEM file, you should verify the content

– Ensure that the certificate chain is complete

– Tip: for an existing keyfile you can use the show command to dump all certs and verify the resulting file

● kyrtool.exe verify c:\domino\all.crt

– Successfully read 2048 bit RSA private key

– INFO: Successfully read 4 certificates

– INFO: Private key matches leaf certificate

– INFO: IssuerName of cert 0 matches the SubjectName of cert 1



– INFO: Final certificate in chain is self-signed

SHA-2 for S/MIME

● For SHA-2 with S/MIME the “FIPS 140-2” algorithms are required

● Enable Option in Person Doc on Admin Tab

– “Can decrypt documents using FIPS 140-2 approved algorithms: YES”

● FIPS 140-2 algorithm needs at least 1024 bit keys

● Many Domino environments still use 512/630 Keys

– This usually leads to increasing Cert.ID,/OU-Cert.ID, Server.ID, User.ID Cert Len in combination with a key-rollover

● This is not a simple click & ready project

– You have to plan this migration!

– There are currently known issues

● Still working with customers to complete their cert/key-rollover project

ID Cert & Key Rollover – How it should work

● First recertify Certs.ID, OU-Cert.ID

● Than recertify servers and users with the higher key len

● Finish re-certification before starting key-rollover for server and later users

– Key-Rollover is triggered in Server doc for servers and Security Policy for users

– Server/Client creates new Private/Public Key and sends a public key sigining request

– Admin uses Certifier or Domino CA zu recertify server/user

– Potential conflict with “Public Key checking”!

– Certificate is always pushed to the user via changed certificate in person/server doc

– Client/Server pickup certificate

– User Workstation will push modified ID changes to ID-Vault

Known Issue for ID-Vault with Recert/Keyrollover● User.ID is overwritten with the ID in ID-Vault

– Recertification and Key-Rollover fails

● Defect SPR # YDEN9KYL23

– Local Id Is Being Overwritten By The Copy In The Id Vault During Rollover/Recertification Even Though Local Id Is Up The issue reported on SPR YDEN9KYL23 was possible to be worked around by Deleting the affected user.id from the id vault and running updall -r on the id vault database

● We are still investigating if this fixed all of our problems for re-certification and key-rollover

– Planning a blog post once I know all details and it completely works

Increasing Internet Certificate Key Size

● Domino 9 Internet CA Supports SHA-2

– You can remove an re-create the Internet Certifier with SHA256 and higher key len

Internet CA Result

● Resulting CA can be used to assign new certificates to users via Person Doc

External Intern Certificates

● There is still no simple way to import external certificates

– User has to manually import the cert

● Supported C-API Call to import the X.509 Certificate

– Possible solution:

● Send the complete X.509 Certificate in a password protected P12

● Send the password in the same email

● Encrypt the email with the Notes.ID

● Call the C-API with following parameters

– STATUS LNPUBLIC PKCS12_ImportFileToIDFile(char *pPKCS12Filename,char *pPKCS12Filepassword,char *pIdFilename,

char *pIdFilepassword,

DWORD ImportFlags,DWORD ReservedFlags,void *pReserved);

Domino Stability and Performance

Domino Transaction Logging

● Domino Translog writes database changes to separate Translog files

– Sequential writes to TXN Files

● Asynchronous “Logasio” process writes changes from translog to Notes Database

● Major differences with Translog

– Database consistency

● If server crashes, transaction log applies pending changes

● Very fast - No consistency check is needed

– Database locking is changed

● Performance might increase because Domino can continue to work on other reuqests while “logasio” writes back changes

– Allows online backup of databases with Notes C-API based Backup API

Transaction Log – Basically two Modes

● Circular Translog

– Database changes are written to Translog just for data consistency/performance

– Usually 4 GB space required

● “Oldest” Translog Data will be overwritten

● Archive Style Translog

– Continuous writes and creation of translog files

– Backup Application (Using the backup API) has to take care to backup and release Translog files

– Usually 20-60 GB space required

● If backup does not run and you are running out of space your server will stop working!

– Can be used to restore a database “point in time”

● Needs Backup Application to support archive translog and “point in time” recovery

Translog Requirement

● Separate Disk with sufficient space

– 4,5 GB for Circular, much more for Archive Style depending on your server

● If you have sufficient cache you don't necessarily need a separate disk either

– But a strong requirement is a cached, low latency write

● Even in a virtual environment translog should be on a separate VMDK file

– Larger VMDK files can cause I/O contention specially with concurrent NSF / Translog read/write operations

Translog Settings● Log Path should point to a separate

“disk”

● Circular Translog and 4GB size

– Unless you have backup solution that supports archive backup

● Automatic Fixup

– Makes a lot of sense

● Runtime/Restart Performance

– Standard is the best balance

● Ouota enforcement

– Not a Translog option but important for quota enabled databases

● Check space used in file when adding note● No compact needed to reduce size for quota

Disk Configuration● Different Type of options depending on your strategy: Local Disks,

SAN/NAS

– But still similar configuration even in virtual environments

– Strong Requirement: “Separate Disks” for

● OS/Swap/Binaries/OS Log etc

● Domino Data (NSF)

● Translog (TXN)● DAOS (NLO)● Optional if needed separate disk for Fulltext Index (FT) New since 8.5.3

– notes.ini FTBasePath=c:\FTBasePath

● “Separate Disk” can mean different things in different environments

– Local Disk = Separate RAID volume

– SAN = Different Logical Unit (LUN)

– NAS = Has it's own file-system and depends on sizing and capability of the NAS

– VMware ESX = separate VMDKs – even if they resist in the same disk volume

Reference - Disk Configuration Details

● Translog needs fast sequential write

– Ensure that write cache is enabled!

– Should be separated from “data” but the only key requirement is low latency write

– Usually a separate RAID1, separate VMDK etc

● Domino Data

– RAID10 instead of RAID5.

– High End SAN hardware should be always RAID10

– Performance mostly depends on

● Cache, Number of disks, Performance of each disk

– Domino Data needs fast/small random I/O

– Average I/O Rate: 0,5 IOPS per registered user

● DAOS

– Needs lower performance and has larger sequential I/O requests

– Could be RAID5 or NAS (even with UNC Path)

– But with UNC path you need sub-directory on NAS side!!!

VMware ESX Virtualization

● Generally a good idea for all smaller servers, test servers, HUB, SMTP, Traveler, servers

● For larger servers with more than 500-800 users all components have to play well together and you should plan for sufficient resources

– Domino can be very resource demanding specially in the I/O area

– Supported even for larger servers since ESX became a “tier-1 virtualization platform”

● Have separate VMDKs for all parts of the server as mentioned before

– OS, NSF, Translog, DAOS, ..

– Take care where VMDKs are located on the SAN side and who you share your storage and servers with!

● Plan sufficient Virtual CPUs

OS Level Antivirus

● Disable Antivirus for the following file types

– *.nsf, *.ntf, *.box

– *.ncf, *.ndk

– *.TXN

– *.DTF

– *.nlo

– Not excluding those files could lead to performance issues and even crashes or hangs

● If you need Antivirus scans on Notes databases you should use a Domino aware Antivirus solution

– Mail Scans on Gateway & Scheduled Scans on Databases

● TIP: Review exclusion lists after every antivirus software (not pattern) update!

Linux File-System Tuning

● Use your favorite journaled file-system ext3, etx4, Reiser FS, …

– RHEL 6.x has Ext4 support (by default) / SLES 11 SP2 does not support Ext4 yet

● Disable write of meta information via mount option -noatime

– Meta information like last accessed time

● A real Runfaster=1 Parameter:

– Change the default scheduler from CFQ (complete fair queuing) to NOOP

– CFQ tries to optimize disk access by reordering requests

– But it would be better to send it to a SAN, RAID controller directly

– Tests have shown that this works better for almost all SAN or local disk configurations

– Dramatical improvement!

– Disable per device

● echo noop > /sys/block/hda/queue/scheduler

– Disable globally via kernel boot parameter

– Edit /boot/grub/grub.conf and enter in kernel line elevator=noop.

●

Reference - Linux Performance CFQ vs noop

● Read-Test - 80 thread to read 32000 docs each

– 80 separate local databases on the server with small documents

● Result:

– 51 sec with CFQ scheduler

– 28 sec with noop scheduler

– 19 sec all data in cache

● Write Test - 80 threads creating 2000 docs each

– 80 separate local databases on the server

● Result:

– 132 sec with CFQ

– 42 sec with noop

● Environment: SLES11 SP2 with local RAID10 disks

● Test-Tool: iostat -x 2

– check the improvement in the “await” column

NSF Buffer Pool

● There have been a lot of discussions about NSF Buffer Pool

– The old rules to have 3/8 of the addressable memory is not helpful any more

– Forget about parameters like PercentAvailSysResources

– The NSF Buffer Pool aka UBM is used for database I/O

● By default the NSF Buffer Pool is 512 MB for 32bit and 1 GB for 64bit

– The default works for almost all configurations

– A cache/buffer is only effective until a certain size

● Check Server Statistics

– Database.Database.BufferPool.PercentReadsInBuffer

– Interpretation: Bad < 90% < PercentReadsInBuffer < 98% < Perfect

– If you really need to change it use notes.ini NSF_BUFFER_POOL_SIZE_MB=n

Database Cache

● Used for caching open databases

– You should have sufficient entries for all databases than can be open on your server!

– Number of users / Number of databases whatever is higher

– –Default: NSF_BUFFER Pool size multiplied by 3

● That's where NSF Buffer Pool comes into play

● Check Server Stats

– Database.DbCache.HighWaterMark

– Database.DbCache.CurrentEntries

– Database.DbCache.MaxEntries

– Database.DbCache.OvercrowdingRejections

● Interpretation

– Good = HighWaterMark < MaxEntries

– Good = 0 OvercrowdingRejections

● Tune: e.g. notes.ini NSF_DbCache_MaxEntries = 3000

Important Server Settings

● Increase Shared Mem Max Size for Domino 32bit on 64bit OS to 3 GB

– ConstrainedSHMSizeMB=3072

● Default for many pools is too small

– EVENT_POOL_SIZE=41943040

– For servers with many Databases

● CATALOG_POOL_SIZE_MB=100● dirman_poolsize_mb=100

● nsf_monitor_pool_size_mb=200

● Separate Update FT Thread and have FT use system memory

– UPDATE_FULLTEXT_THREAD=1

– FTG_USE_SYS_MEMORY=1

● Router Optimization

– RouterMaxConcurrentDeliverySize=1048576

– Disable_BCC_group_expansion=1

NSF Optimization

● Use the current ODS on Clients and Servers

– Create_R85_Databases=1 / Create_R9_Databases=1

– TIP: Use client desktop policies to upgrade ODS on Notes Clients!

● Enable Data and Design Compression

– Compact -C -v -n

● Take benefit of DAOS!

– Reduces disk I/O, disk space, backup time (incremental backup), increases scalability

– Put DAOS on different disk

● Rebuild Views after moving to new ODS and mailfile-design

– Updall -r

– Will optimize view collections (only first sort collation will be build)

Disable Unread-Marks if possible

● Unreadmarks need to be maintained per user

– Will be updated when the user opens a database

– If you have many changes in the database since last open it will take some time!

● But this is not the only reason to disable unread marks...

– Some operations take much longer if unreadmarks are enabled

● One critical issue:

– Even if you have concurrent web-agents enabled, only one agent will run at a time because unread marks need to sync after each agent run!

● Strongly recommended: Disable Unreadmarks if possible

● TIP: You should disable unreadmarks for all system databases if you don't need it

– names.nsf, log.nsf, mail.box, admin4.nsf, statrep.nsf

Advanced DB Properties(Last Tab)

● ON/OFF (depending on app): Don't maintain unread marks

● ON: Replicate unread marks: Never, Cluster servers only, All servers

● ON: Optimize document table map – specially when using form = “..”

● ON: Don't overwrite free space

● OFF: Maintain LastAccessed property

● Maybe ON: Don't support special response hierarchy

– needed for @AllChildren and @AllDescendants

● ON: Use LZ1 compression for attachments

● ON: Allow more fields in database (compact -K / -k)

● ON: Allow soft deletions

● Maybe set to low value: Limit entries in $UpdatedBy / $Revision fields

Domino 9 Clustering

Clustering - Introduction

● Why Clustering?

– Email and Domino Applications are mission critical for most customers

● Needs to be available during business hours● Many customers even need 24x7 availability

– It's not just about the TCO of your servers. It's more about the cost of not having a service available when you need it

● High availability for Domino Servers

– Better defined as High Availability of Domino Service not just the individual server

– The total availability depends on

a.) Planned Down-Time and b.) Unplanned Downtime for your Domino Service

● You cannot reduce both to zero

● You should avoid all kind of Single Point of Failures (SPOF) for mission critical services

– A single point of failure is a anything that is not redundant enough and whose failure will cause damage to the availability of a service

It's not just about your servers but also about all other resources like network, switches, data centers, ...

Difference between Domino and other High Availability Solutions ● Most solutions only use a “stand-by” servers

– SAN is mirrored to a different data center and stand-by server is started with the same name and IP address when the other server or data center fails

● Domino works completely different

– You have separate servers with own name, own IP address and own storage

– Data is almost “real-time” replicated between servers

– Both servers are online and can actively used – load-balancing

● Domino Clustering provides protection against

– Disaster Recovery (when a whole data center goes down)

– Server failure

– Failure of a single database

– Server crashes

– Database Corruptions (usually corruptions do not replicate)

– Shut down servers for maintenance

Domino Clustering

● Clustering is available since Notes/Domino 4.5

● Two or more completely independent servers are put together into a „logical“ cluster with a logical cluster name

● Notes Clients are cluster aware and will connect to another cluster if the current server fails

● You can mix different versions and platforms

● Components of a Domino Cluster

– cldbdir.nsf and cldbdir task - Cluster directory and task

– clubusy.nsf - Cluster Busy Database

– Cldbdir - Cluster directory task

– Clrepl - Cluster replicator servertask

– Cluster admin task and cluster manager (now integrated into the server)

– cluster.ncf (file on notes client containing the cluster-names and cluster servers)

– Admin Client Panel for Cluster Administration

–

Cluster Tasks

● Cluster Admin and Cluster Manager

– Integrated into the server

– Used to maintain the cluster

● Cluster DB Dir Servertasks

– Automatically started

– Maintains cluster directory

● Cluster Replicator

– Used for cluster replication

– By default one cluster replicator is started

– If needed you can add multiple cluster replicators

● In earlier releases you had to start multiple cluster replicators cia servertasks=● notes.ini cluster_replicators=n

How Cluster Replication Works

● Document changes are captured and trigger the cluster Replicator via a message queue

● Cluster Replicator reads message queue and pushes changes to other all other replicas in the cluster regardless of replication settings (aka almost "real time" replication)

Cluster.ncf used for Failover

● Cluster.ncf located in data directory of Notes Client

● Automatically updated when client connects to a cluster

● Used to connect to another server if the own server is down

– Client contacts to another server in the cluster

– Server checks in cluster directory for another replica for the database

– Client is redirected to most available server for the replica

Time=30.07.2014 16:44:23 (C12577D0:005675FC)nsh_cluster85CN=nsh-d85-win-01/OU=Srv/O=NashComLabCN=nsh-d85-win-02/OU=Srv/O=NashComLab

Client Failover Improvements

● Notes 8.x provides enhancements for failover

– You are prompted to select another replica when the server goes down

– Notes.ini Settings allow to automatically failover

● HidePromptFailoverInc=1● Replaced by new functionality in 8.5.2

–

– New Notes 8.5.2 silent failover option to completely disable failover messages

● Needs a desktop policy setting enabled● Sets notes.ini FailoverSilent=1

–

What the Cluster Replicator will not do

● The cluster replicator will not initialize a replica stub

– Scheduled replication (or manual replication) is required to populate the new database.

– Better option is to use accelerated replica creation

● The cluster replicator / SCR will not honor selective replication settings nor honor readers field before transferring documents

● No failover if the server hangs and still somehow responds

● Cluster replicator does not record every successful replication event

– Individual replication events are stored in memory, and later recorded in the Replication History as a single event.

● The cluster port will not failover by default if cluster port goes down

– Workaround: Server_Cluster_Auxiliary_Ports= <portname> or better * for all ports

● Agents will not failover to a different server automatically

– Good news Out of Office is now integrated into the mail router and needs no agent anymore

Known Failover Issue

● Fail-Over causes server “hang”

– With Failover in a cluster there is a large impact when the clients do a ServerGetClusterReplicaMatches call to find the server to fail over to. The impact can be so large as to make the server unavailable causing existing clients connected to failover, bringing the entire cluster down

● Regression introduced in 8.5.2 FP2

– Finally fixed in 8.5.3, can be enabled in 8.5.2 FP3 via notes.ini DISABLE_CLUSTER_MATCHES_FAST=1

– Reference: SPR# SWAS8GGHMC

● The Cluster Server needs to find a replica for a database via C-API call “ServerGetClusterReplicaMatches”

– That call had a performance issue when called many times at once

● Resource contention

● This fix resolves the issue

Windows 2008 Port Stealth Mode Cluster Failover Issue● 20 Second Failover Delay

– Windows will not reject packages when application port is down

– Client will not know that the application is dead and continue to wait – usually 20 seconds

● From IBM Technote #1498755

– Windows 2008 has a feature called "Stealth mode" wherein by default it does not respond to network requests on ports for which no process is listening. This feature creates a problem with the Notes client failover when a server is down, since the OS does not respond to the client and thus the client will not failover until it's timeout has been exceeded. Although Domino does support Windows 2008, at the appropriate version level, this is a feature which "breaks" the Notes client failover.

● The following link gives directions by Microsoft on how to disable "stealth mode"

– http://msdn.microsoft.com/en-us/library/ff720058%28v=prot.10%29.aspx

Workload Balancing

● Server Availability Index (SAI) is used to determine the work-load

– Based current response time of the server

– SAI from 0 to 100 where 100 is 100% available

● Availability Threshold is used to trigger failover

– Set config Server_Availability_Threshold = n in %

● Below that level users will failover to another server if that other server is „less busy“

– Usually it makes only sense to set a very low value to ensure users failover if the server has really bad performance

● Split the load logical e.g. By home-server

● SAI was broken before Domino 8.5 and does now work a lot better

– But it needs “configuration” on fast servers.Detailed Slides in the Appendix

High Availability for Internet Protocols● ICM – Internet Cluster Manager could be used to distribute load

and high availability of Domino HTTP services

– But you need a separate server running ICM

– ICM could also be a Single Point of Failure

– Making ICM high available needs a second ICM and a Load Balancer

– So why not using a Load-Blancer directly

● Most customers use Load-Balancer or Secure Reverse Proxy

● You don't need a separate solution for Domino and can leverage what you company already has

● Many secure Proxies already have high-availability options

“Load Balancer”

● With two or more servers we need component in front of it to distribute the requests

● Different types of solutions

● Load-Balancer/IP Sprayer

– IP based solution distributing/dispatching IP sessions on lower network layer

– Example: Websphere® Edge-Server, Many Appliances

● Secure Reverse Proxy with Load-Balancing Module

– Traffic is “forwarded” to multiple back-end servers with separate IP addresses

– Example: IBM Mobile Connect, Open-Source solutions like Apache with additional modules, Pound, Many Appliances

Example for Session Stickiness

● Restaurant

– Guest (Web Browser)

– Talks to the Supervisor (Load-Balancer)

– And gets an Waiter (Server) assigned to serve the guest based on Waiter availability

– The user will stick with the waiter until the waiter is unavailable

– In that case the guest will go back to the Supervisor and asks for another Waiter

● This works as long you can define a central supervisor

– But what happens if you don't control the whole environment or your supervisor is a Single Point of Failure?

– For example in the Internet you only own your environment

– And you can create certain DNS settings

– You can define a multi-homed Server or Supervisor

– Browsers will use one of the addresses to access any of the Servers / Load Balancers based on availability, location, latency etc ...

– Check nslookup www.google.com for an example

●

http://www.google.com/

Secure Reverse Proxy

● More flexible solution

– Filter requests only allow for example /traveler requests or iNotes request etc.→– Redirect HTTP to HTTPS requests

● Only allow certain user agents

● Terminates IP Connection in DMZ

– New IP connection from Reverse Proxy to back-end servers

– You can off-load SSL Traffic

– Usually network between Reverse Proxy and Traveler Servers is trusted / controlled by firewall

● Depending on your solution you can already authenticate the user

– Via LDAP against Domino directory

– Depending on the solution the basic authentication header (user/password) is passed to Traveler

– Some solutions like IBM Mobile Connect can create LTPA Session Cookies passed to Traveler

Example - POUND - REVERSE-PROXY AND LOAD-BALANCER● Reverse proxy, Load Balancer and HTTPS front-end for Web

server(s)

– Distributed under the GPL included for example in OpenSUSE

● Very small and easy to configure

– “Pound is a very small program, easily audited for security problems. It can run as setuid/setgid and/or in a chroot jail. Pound does not access the hard-disk at all (except for reading the certificate)”

● Can be configured in a couple of minutes

● Reference

– Wikipedia: http://en.wikipedia.org/wiki/Pound_(networking)

– Homepage: http://www.apsis.ch/pound/

POUND - REVERSE-PROXY AND LOAD-BALANCER● Pound does already what we expect from a normal “Secure”

Reverse Proxy

– Reverse-Proxy: Passes requests from client browsers to one or more back-end servers

– Load Balancer: Distributes requests from the client browsers among several back-end servers, while keeping session information.

– SSL wrapper: Decrypts HTTPS requests and pass them as plain HTTP to the back-end servers

– Fail Over-Server: Should a back-end server fail, stop passing requests to it until it recovers.

– Request Redirector: Requests may be distributed among servers according to the requested URL

● Restrictions:

– Cannot connect to HTTPS back-ends – so it is only “secure” for internet facing traffic

– Cannot authenticate users so no way to check authenticatio retries at gateway level→

Configure Pound for two Back-Ends

● Basic configuration

– Listens on HTTPS (443) IP 192.168.100.151

– Allows “Options” requests (xHTTP 2)

● Note: Specially needed by Traveler

– Uses our SSL certificate we just have created

– Only allow Traveler Requests

– Same priority for both HTTP Backends

● 192.168.100.161, 192.168.100.166

– Session Stickyness based on Authorization header

– TTL for the idle session 1 hour

● Configure and start via

– vi /etc/pound.cfg

– /etc/init.d/pound start

● Done!

ListenHTTPS Address 192.168.100.151 Port 443 CheckURL ".*" xHTTP 2 Cert "/etc/ssl/wildcard.nashcom.loc/host.pem" Service "traveler" URL "^(/traveler|/servlet/traveler).*" BackEnd Address 192.168.100.161 Port 80 Priority 5 End BackEnd Address 192.168.100.166 Port 80 Priority 5 End Session Type BASIC TTL 3600 End End End

iNotes Redirect Database

● The IBM iNotes Redirector

● Acts as an entry point and authentication prompt for IBM iNotes products

● Database contains configurable settings for redirection, SSL, customization and mode selection/availability.

iNotes Redirect Database

● Create database and start configuration

● Ensure database has “No Access” for Anonymous user

– This will trigger authentication before redirect

● Redirect will lookup the user and redirect user based on configuration

● Multiple ConfigurationTypes and Options available

Redirection Types

● 3 Redirection Types

– Fixed

– Dynamic

– MailServer

How does this work with multiple Clusters?

● One set of servers as entry point for your iNotes Redirect DB

● Choose “Mail-Server” redirection option

● One approach

– Use a different “Full Qualified Server Name” than Domino Server Name because name need to point to the proxy instead of the server itself via DNS

– Ensure the name points to your proxy

– Have the primary back-end server in the proxy for that entry point to the primary server

– Have the cluster partner as a fall-back server

– Do the same for all your mail-servers

– That way iNotes Redirect will always send users to their home-mail-server

● Failover occurs only when home-mail-server is not reachable

REF: Many very detailed Options

REF: UI Options

REF: Mobile Device Options

Domino Web Server Configuration DB

● DomCfg.nsf is required for using the custom login form

Map Redirect Form

● Create a “Sign In” Form Mapping

– Point to DWALoginForm in your redirect database

Redirect Login Form UI

● This is the default UI

● You can customize the UI

– Only the Look & Feel but not the fields!

– Most of the internal logic is hardcoded in the HTTP Task

Single Sign On (SSO) for HTTP

● In case of a failure (or redirect) you don't want to authenticate again

– You should use multi Server session authentication on all Domino Servers for security reasons anyway

– “Basic authentication” would send username + password base64 encoded with every request

– Single Server has some limitations and should not be used

– Best supported is Multi Server Session Authentication using LTPA cookies

SMTP High Availability

DNS Settings for SMTP Failover

● Multiple MX records are used for multiple inbound hosts

– Well known configuration that does not need a load-balancer

– Also works for configuring outbound relay hosts

● Define a virtual Domain Name and configure this name in server config document as your relay server

● The router will use it based on the preferences you define● Alternate Option: Load-Blancer with a service / virtual IP for multiple servers● If your Antispam Solution needs a lookup for existing mail-addresses either specify

multiple LDAP Servers or use a Load Balancer

nslookup> set query=mx> iris.comServer: nsh-dmz-01.dus.nashcom.deAddress: 192.168.96.20

Non-authoritative answer:iris.com preference = 10, mail exchanger = capricorn.iris.comiris.com preference = 20, mail exchanger = arista.iris.comiris.com preference = 40, mail exchanger = elektra.iris.comiris.com preference = 30, mail exchanger = epic.iris.com> quit

Traveler Server HA

● Traveler itself is “cluster-aware”

– Uses the cluster directory to check where users have their mailfiles

– Fails over automatically by switching to a different replica in the mail cluster

– Works for Standalone and Traveler HA

● Traveler HA

– Uses relational backend DB for state information (DB2 / SQL Server) instead of local DB

– Provides fail-over and even more important scalability improvements

– You can have many servers in a Server pool

● Traveler leverages the HTTP task

– You have to have a load-balancer in front of your Traveler server

– HTTP OSGI Servlet will connect to Traveler servertask on the same or different server based on load

REF: Failover and the IMAP server

● New in 8.5.1

● TN #1429885

● notes.ini IMAP_UIDVALIDITY_ROOT=<n>, where <n> is a decimal number from 0-255

● "Helps" when IMAP fails over

● You still need a Load-Balancer in front of IMAP

Browser Plugin

Notes Browser Plug-in

● Run Notes applications from a “browser-plugin”

– “No” modification needed. Runs “all” classical Notes Applications (Basic Client features)

● Not designed to support mail template

– Designed to work in conjunction with iNotes

– Provides a compliment to existing XPages capabilities

● Windows only (for now, at least!)

● Firefox and Internet Explorer along with Citrix, in first release

● Plugin uses most of the Notes Basic Client Libs and needs a data directory with notes.ini, notes.id and system databases

– But you can leverage ID-Vault and policies to configure the underlying client configuration

● You still need a Notes Client License

– Enterprise CAL, CEO Communications, Collaboration Express

●


● Operating Systems Supported

– Windows® XP, Windows® 7, Windows® 8

– Citrix

● Browsers Supported

– Firefox (version 17 ESR & higher)

– Internet Explorer 8 and higher

● Feature Details

– Migration of Notes bookmarks to Browser Favorite bookmarks

– Access to Local or Server Replica and databases

– Support for Roaming Users, Desktop Policies, Replication

– Email integration with iNotes

– Co-existence with Notes Standard Client

● Plug-in Installed as part of standard client install

● Cannot be run at the same time


● Common URL's supported by the browser plugin:

– notes:home -- launches home/welcome page

– notes:workspace -- launches Notes workspace

– notes:replication -- launches replicator tab

– notes:///names.nsf -- opens local names.nsf (name & address book)

–

● Browser Plugin in Passport Advantage

– IBM Notes Browser Plug-in V9.0.1 Fix Pack 2 for Windows English with JVM (CN0YGEN )147mb

● Full Version incl. Java and Viewers

– IBM Notes Browser Plug-in Lite V9.0.1 Fix Pack 2 Windows English without JVM (CN0YHEN ) - 55 MB

● If plugin doesn't start, see if it's enabled in the browser:

Browser Plugin Example

Notes Browser Plugin Futures

● Futures from IBM Connect Presentation

– Sametime integration

– Single-Sign-On

– Chrome support

– MAC support

– Web install/update

– Auto Update

– Offline and Archiving Support

– more.....

Q&A

● Thanks for your attention!

● I hope you enjoyed the session

● Questions now or later?

● Contact

– http://blog.nashcom.de

– http://www.nashcom.de

– [email protected]

– +49 172 2141912

● Bonus Slides at the end of the presentation

Appendix

TLS Support via new IBM HTTP Stack● TLS is required for higher security in some customer environments

– TLS stands for an encryption standard next higher level above SSL 3.0→– And also stands for SMTP Channel encryption but that does not mean necessarily that the →

TLS chipers are used for TLS in SMTP

● IBM implemented TLS for HTTPS only using the IBM Webserver in front of a Domino server

– Only supported for Window in the 9.0/9.0.1 release

– Install Option will install into a separate directory below the Domino Binary location

– Domino starts IBM HTTP webserver (based on the Apache webserver)

Enable IBM HTTP Server

● Notes.ini HTTPIHSEnabled=1

– Enables Startup for IBM HTTP Server

● This setting changes the Domino HTTP server to behave as follows

– Disables the usual ports configured in the Domino Directory

● e.g. HTTP port 80 and the HTTPS port 443

– Overwrites Domino HTTP server connection settings settings that maximize the re-use of connections between mod_domino/IBM HTTP Server and the Domino HTTP server.

– By default, the Domino HTTP server listens on port 9288 for loop back connections from mod_domino/IBM HTTP Server.

– Domino HTTP server only accepts connections that originate from the same computer.By default, mod_domino uses the local loop back address of 127.0.0.1 to connect to the Domino HTTP server. Both server processes must run on the same computer.

● SSL/TLS configurations

– Use iKeyMan Application to manage certificates

Setting up a Cluster

● Very easy and straight forward

● In 8.5.3/9.x setting up a cluster works online without a restart

– Demo Setting up a Cluster

● You still need to enable a couple of settings and setup a separate cluster port

– In earlier releases it was recommended to use a separate network card for the cluster port

– Many customers used cross-over cable for the cluster port

● Does not work with separate data-centers

– Currently a gigabit connection is sufficient fast for user, server and cluster traffic

– You should still have separate IP addresses and Notes Ports for the cluster traffic

● Because servers connecting use separated dedicated server thread-pool threards● Cluster port should not be compressed and encrypted if possible

– You can still use the separate NIC to team your network cards for performance and failover

● Or to have a dedicated network for backup traffic

Developing Cluster enabled Applications

● Never use NoteID always use UniqueID (UNID)

● Open databases with failover (in Lotus Script, Java, C-API)

● Use "Unprocessed Documents" only for performance reasons

– use a field in the document to check the status instead!

– you never know if an agent might need to run on a different server or the agent-design changes...

● Don't use hard-coded server or database names if possible

● Don't assume to find databases in the same directory on the same server

● Have a reasonable "on error handling" for catching errors and resume

– You cannot always recover automatically if a server goes down

● Scheduling agents for failover is extremely difficult - you might have to make some trade-offs ...

Notes.ini Network Parameter

● Bind IP address to Domino partition

– TCPIP_TcpIpAddress=0,192.168.96.111:1352

– SMTPNotesPort=TCPIP

– POP3NotesPort=TCPIP

– LDAPNotesPort=TCPIP

– IMAPNotesPort=TCPIP

● Have an additional port for cluster

– Server.Doc and notes.ini settings

– TCPCLUSTER_TcpIpAddress=0,192.168.96.111:1352

– SERVER_CLUSTER_DEFAULT_PORT=TCPCLUSTER

Cluster Status

● „Show cluster“ provides information about the cluster status

> sh cluster

Cluster information:Cluster name: nsh_cluster85, Server name: nsh-d85-win-01/Srv/NashComLabServer cluster probe timeout: 1 minute(s)Server cluster probe count: 740Server cluster default port: *Server cluster auxiliary ports:Server availability threshold: 0Server availability index: 100 (state: AVAILABLE)Server availability default minimum transaction time: 3000Cluster members (2): Server: nsh-d85-win-01/Srv/NashComLab, availability index: 100 Server: nsh-d85-win-02/Srv/NashComLab, availability: UNREACHABLE

Working with clusters / Administration

● Admin Client support for clustering

– Set single Dbs out of service etc

● Cluster Analysis tool

● Server statistics

– Specific statistics for clustering

● Ensure all Servers have Manager Access and all Roles in all databases

● Ensure an hourly replication document is enabled

– Maybe even have a startup replication defined?

● Debugging / Logging

– Log_replication=1

– RTR_Logging=1

Statistics – (Cluster) Replication

● Use to check Cluster Replicator Performance

● Check Server Stats

– Replica.Cluster.Failed

– Replica.Cluster.SecondsOnQueue

– Replica.Cluster.WorkQueueDepth

● Interpretation

– Perfect < 10 < SecondsOnQueue > 15 > Bad

– Perfect < 10 < WorkQueueDepth > 15 > Bad

● Tune:

– Add more cluster replicators -> notes.ini cluster_replicators=n

– optimize cluster server usage (e.g. Split active users between cluster mates)

Restrict access to a server

● You can restrict a server for maintenance

– Different options – some have been added and are unknown

– Dynamic setting - Use Set config Server_Restricted= …

● Server_Restricted=1

– No new opens are allowed

– Existing opens still work

● So you should use “Drop all” to terminate existing sessions

– Administrator can connect using remote console

– The restricted server will be able to initiate replication with other servers!

– Other servers will not be able to initiate replication with the restricted server!

– The server will be able to route existing mail in its mail.box(es) for transfer or delivery

– Other servers (without manager access) will not be able to route mail to the restricted server

– Is set back to 0 (unrestricted) when the server is recycled.

Restrict access to a server


– Has all attributes of Server_Restricted=1

– But recycling the server does not change the server state.


– Has all attributes of Server_Restricted=1plus starting with 8.5.2 Fix Pack 3 and 8.5.3this blocks all replication that is not coming from an ID for an administrator. This blocks the local replication of mail databases from Notes clients.


– Has all attributes of Server_Restricted=3But recycling the server does not change the server state.

Agents in a Cluster

● No Out of the box simple solution

● No automatic failover

● Prerequisites

– Use „Unprocessed Documents“ only for performance

– Have separate status field or folder for document processing status

– Reprocessing a document should never process it again

– This is also important when updating agents (unprocessed could be lost – agent data issues)

● But strong requirement for cluster failover

● Different strategies to recover

– None of them is simple and there is some effort involved

– a.) Schedule agents on all servers and put in your own logic to run the agent only on one seerver

– b.) Put something in place to switch agents to a different server

– c.) Practical approach: Most times you have only short server down time. Identify your most important agents and switch them manually to the cluster mate if needed

You still need a plan and take care of what agents need to failover!!!

Changing SAI Settings

● „Show cluster“ provides information about the cluster status

sh ai

Range XF Hits Min AI Max AI1 2 1469 100 1002 4 5 75 803 8 8 55 584 16 11 39 485 32 5 23 286 64 07 128 08 256 09 512 010 1024 011 2048 012 4096 013 8192 014 16384 015 32768 016 65536 017 131072 018 262144 019 524288 020 1048576 0Current value of SERVER_TRANSINFO_RANGE is 6.No change suggested for SERVER_TRANSINFO_RANGE.

SAI and LOADMON

● Domino uses a module called "LoadMon"

– Routine calculating speed of 12 selected transactions

– Checks current transaction performance, summarizes and compares them with previous intervals and minimum values (RunningAvgTime & MinAvgTransTime)

– Unit: microseconds

● OPEN_DB

● OPEN_NOTE

● CLOSE_DB

● DB_INFO_GET

● DB_REPLINFO_GET

● GET_OBJECT_SIZE

● READ_OBJECT

● GET_SPECIAL_NOTE_ID

● DB_READ_HIST

● DB_WRITE_HIST

● SERVER_AVAILABLE_LITE

● NIF_OPEN_NOTE

Expansion Factor (XF)

● XF is calculated based on the performance values of current transactions in relation to minimum time for a transaction

– It's the number of times the current transactions take longer than the minimum transaction time

– XF values for different transactions build a overall XF

– This XF is computed and converted into AI based on a Range to scale the XF

● SAI is calculated based on XF and the transinfo range (n)

● SAI = 100 * (1 - log (XF) / log (2) / n)

– Notes.ini Server_Transinfo_Range n is 6 by default and specifies the maximum Expansion Factor of a Domino Server. The XF is calculated 2 raised to the power n (64 by default)

Issues with SAI and LOADMON

● SAI was broken until Lotus Domino 8.5

– There have been a couple of issues with LOADMON. The last one known has been fixed in D8.5

● SAI calculation on fast servers still might not work for you out of the box

– LOADMON uses micro seconds

● On a fast server at idle times transactions can take only a couple of micro seconds● Compared to normal performance e.g. 1 ms this can result in very high XF● Causes a low SAI for normal performing servers

● Tuning: D8.5 Set range of minimum and maximum values

– notes.ini: Server_MinPossibleTransTime=1500

– notes.ini: Server_MaxPossibleTransTime=20000000

– Important: Delete loadmon.ncf when server is down to delete old minimum values

More LoadMon Notes.ini Settings

● SERVER_TRANSINFO_MAX (default 5 / max 60)

– number of statistics collections stored in LoadMon

● SERVER_TRANSINFO_UPDATE_INTERVAL (default 15)

– interval for statistics capturing & calculation

● SERVER_MIN_TRANS (default 5)

– minimum transactions needed for a statistic value to be valid

● SERVER_TRANSINFO_NORMALIZE (default 3000)

● SERVER_TRANSINFO_HTTP_NORMALIZE (12000)

– used to initialize empty statistics (zero in loadmon.ncf) on startup in Domino

Debugging LoadMon

● debug_loadmon=1

– Enables LoadMon Debugging, writes additional information to server console

– Loadmon: Domino AI = 100, XF = 1

– Adds additional 46 statistics counters (server.loadmon.*)

● loadmon.ncf

– loadmon.ncf in Domino data directory stores last information from loadmon before server is shutdown

– loaded on server start to initialize statistics counters