Managing Enterprise Directories: Operational Issues

Performance Monitoring

Brendan Bellina, University of Notre DameBase CAMP – Tempe, ArizonaFebruary 5-7, 2003

Copyright Brendan Bellina, 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Base CAMP - February 5-7, 2003 Middleware: Directories 2

How to present in 33 minutes (or less!)

• No humor• No arbitrary anecdotes• No incoherent rambling• … but how will we know it is a middleware

presentation?

That is why there is a page footer.

One more thing…

Volunteer alarm clock

Base CAMP - February 5-7, 2003 Middleware: Directories 3

Presentation Overview

• Why Monitor Performance• What Performance Metrics To Monitor• How To Gather Performance Metrics• Tools for Performance Monitoring @ ND

– Orca– Orcallator– Look

Base CAMP - February 5-7, 2003 Middleware: Directories 4

Why Monitor Performance

• Availability

It’s up… isn’t it?– Directories often require 7x24 availability

• Responsiveness

It’s fast enough… maybe.– Directories often require extremely fast response

• Scalability

We can handle that… I think.– Structural changes, indices, increases in the # of entries or #

of attributes can all affect performance.

Base CAMP - February 5-7, 2003 Middleware: Directories 5

What Performance Metrics To Monitor

Response time

Connection requests

Bind requests

Bind errors

Search requests

Search errors

Average count & size of search results

Directory Cache Hits

Directory Cache Tries

Bind response time

Search response time

Current connections

Avg connection length

Current binds

Current searches

# Bytes transmitted

# Entries transmitted

Base CAMP - February 5-7, 2003 Middleware: Directories 6

How To Gather Performance Metrics

Directory internal query (iPlanet, OpenLDAP)base search on cn=monitorldapsearch -h directory.whatsamattau.edu -b"cn=monitor" -s base objectclass=*

objectclass=topobjectclass=extensibleObjectcn=monitorversion=Netscape-Directory/4.16 B01.300.2035threads=30currentconnections=27totalconnections=118499dtablesize=1024readwaiters=0

Base CAMP - February 5-7, 2003 Middleware: Directories 7

How To Gather Performance Metrics

base search on cn=monitor cont.opsinitiated=410383

opscompleted=410382

entriessent=173812

bytessent=59522106

currenttime=20030123212714Z

starttime=20030123115756Z

nbackends=1

dataversion=directory.whatsamattau.edu:389 020010212195544 0

ldapserverconfigdn=cn=ldap://:389,dc=directory,dc=whatsamattau,dc=edu

backendmonitordn=cn=monitor,cn=ldbm

Base CAMP - February 5-7, 2003 Middleware: Directories 8

How To Gather Performance Metricsbase search on cn=monitor, cn=ldbmldapsearch -h directory.whatsamattau.edu -b"cn=monitor,cn=ldbm" -s

base objectclass=*

objectclass=top

objectclass=extensibleObject

cn=monitor

database=ldbm

readonly=0

entrycachehits=505559

entrycachetries=540329

entrycachehitratio=93

currententrycachesize=10000

maxentrycachesize=10000

dbchehits=1011415

dbcachetries=1043689

dbcachehitratio=96

dbcachepagein=32268

dbcachepageout=851

dbcacheroevict=20160

dbcacherwevict=0

dbfilename-0=wuAttribute_1.db2

dbfilecachehit-0=26

dbfilecachemiss-0=4

dbfilepagein-0=4

dbfilepageout-0=4

Base CAMP - February 5-7, 2003 Middleware: Directories 9

How To Gather Performance Metrics

• LDAP log processing– Read the log files looking for events– Calculate statistics

• Directory Probing– Ping– Query to directory service– Bind to directory service– SNMP

Base CAMP - February 5-7, 2003 Middleware: Directories 10

How To Monitor Performance Metrics

• NMS (Network Management System)– HP OpenView– CA/UniCenter TNG– IBM NetView

• Operating System Tools– Win NT Event Viewer– W2K WMI (Windows Management Instrumentation)– W2K Network Monitor; W2K System Monitor– Sun Solaris SE Toolkit

Base CAMP - February 5-7, 2003 Middleware: Directories 11

Tools for Performance Monitoring @ ND

• Orca - http://www.orcaware.com• Orcallator (collector for Solaris statistics)• Look – LDAP operational Orca “k”ollector

Base CAMP - February 5-7, 2003 Middleware: Directories 12

Orca (http://www.orcaware.com)

About Orca:• Generates web friendly performance graphs based on collected

statistical data.• Free (will take donations)• Open Source• Perl• Used by Yahoo!/Geocities• Used at Notre Dame on 100+ servers• Hourly, Daily, Weekly, Monthly, Quarterly, Yearly, All• Did I say free?

Base CAMP - February 5-7, 2003 Middleware: Directories 13

ORCA @ Notre Dame

• CRON job runs every 15 minutes to regen graphs• Orcallator (SymbEL script included with ORCA)

– Written to monitor web servers and Solaris stats– Requires Sun Solaris SE Toolkit (free from http://

www.setoolkit.com)– Runs on each directory server to collect statistics – 5 minute

averages, 30 second peaks– Data is consolidated on one server for ORCA processing– Modifications by ND to read iPlanet Dir Server 4.16 log

Base CAMP - February 5-7, 2003 Middleware: Directories 14

ORCA @ Notre Dame

• ND Modifications- LDAP connection rate- LDAP Bind rate- LDAP Bind error rate- LDAP search rate- LDAP search error rate- LDAP search results returned rate- Picture is worth 1000 words:

- Go to <http://eds.nd.edu/perfmon>

Base CAMP - February 5-7, 2003 Middleware: Directories 15

Look – LDAP operational Orca "k"ollector

• LDAP supplement to Orca and/or Orcallator• Perl script• Produces Orca compatible output files• Processes iPlanet 4.16 log file• Queries directory cn=monitor• Queries directory cn=monitor, cn=ldbm• Beta 0.90 version currently available at

<http://www.nd.edu/~bbellina/downloads/Look>

Base CAMP - February 5-7, 2003 Middleware: Directories 16

Look – LDAP operational Orca "k"ollector

Connection Rate

Bind Rate

Bind Error Rate

Search Rate

Search Error Rate

Search Results Returned Rate

Connections

Operations Initiated

Operations Completed

Entries Sent

Bytes Sent

Entry Cache Hits

Entry Cache Tries

Cache Hit Ratio

Database Cache Hits

Database Cache Tries

Database Cache Hit Ratio

Peaks and AveragesView live results at <http://eds.nd.edu/perfmon/Look>

Base CAMP - February 5-7, 2003 Middleware: Directories 17

ORCA Look Graph Example #1

LDAP search rate before and after email routing was transferred from email-specific directory to EDS:

Base CAMP - February 5-7, 2003 Middleware: Directories 18

ORCA Look Graph Example #2:Troubleshooting

Daily abuse of open LDAP connections.What is different about the weekend?

Base CAMP - February 5-7, 2003 Middleware: Directories 19

ORCA Look Graph Example #3:Troubleshooting

How long has this been going on? Week 49.What happened in mid-December? Hmmm…

Base CAMP - February 5-7, 2003 Middleware: Directories 20

ORCA Look Graph Example #4:Troubleshooting

The culprit has been identified and shot.

Base CAMP - February 5-7, 2003 Middleware: Directories 21

ORCA Look Graph Example #5:Troubleshooting

The sky is blue, the birds are chirping, and all is right with the world.

Base CAMP - February 5-7, 2003 Middleware: Directories 22

LinksNotre Dame Enterprise Directory Service

http://www.nd.edu/~eds

Notre Dame EDS Performance Statistics - Orcallator

http://www.nd.edu/~eds/perfmon or http://eds.nd.edu/perfmon

Notre Dame EDS Performance Statistics – Look

http://www.nd.edu/~eds/perfmon/Look or http://eds.nd.edu/perfmon/Look

ORCA software

http://www.orcaware.com

Look software

http://www.nd.edu/~bbellina/downloads/Look

Brendan Bellina (mailto:BBellina@nd.edu)

<http://www3.nd.edu/~eds/cgi-bin/nd_ldap_search.pl?ldapfilter=uid=bbellina>

Base CAMP - February 5-7, 2003 Middleware: Directories 23

Copyright Statement

Copyright © Brendan Bellina, 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.