magee fixing app performance2008

2008 MartinScott Consulting LLC. All rights reserved.

Fixing Application Performance Problems

Jamie MageeMartinScott Consulting

Who is Jamie Magee?

Irish Co-founder of MartinScott ConsultingFrequent speaker for Advisor, The View, Sphere

Developing Lotus apps since 1994Designing large apps so they scale wellSpeeding up slow applications

Creator of:NoteMan Toolbar for Developers/AdminsfA more powerful Document Properties and more!

WirelessMail for Domino mail to any device Blog: JamieMagee.com

3 things you need to know first

1. All the hardware in the world will not fix a poorly designed application Most performance issues are best resolved in the design of

the applications 2. HTTP server is a Web wrapper around the Domino

server Most Notes application performance tuning concepts still

apply on the Web3. Performance Tuning has trade-offs You can only have 2 out of 3f Good - (functionality)f Inexpensive - (development and maintenance effort)f Fast - (performance)

What Well Cover

Coding for performance Managing view indexing activity Web application performance techniques Configuring server settings for application performance Tracing performance problems Monitoring server resources Wrap-up

Getting a Handle to a View

NotesDatabase.GetView(viewName) Can be slow (e.g., 1+ second)! Fastest implementation is when parameter is the true view

name not the alias f Using correct case!

Using the view alias is easier to maintain, but slower Dont get the same view inside a loop

Tradeoff

LotusScript and JavaLotusScript and Java

Traversing a View or Doc Collection

When writing data to multiple documents, must use LotusScript/Java For traversing a NotesView:

f GetNthDocument(N) a little faster than GetFirstDoc/GetNextDoc

For traversing NotesDocumentCollection:f GetFirstDocument() with GetNextDocument(doc) much

faster than GetNthDocument() When only reading data from documents @DbLookup/@DbColumn is fastest


Reading Values on a Doc from a View

WORKSStatus = doc.Status(0)

BETTER!Status = doc.ColumnValues(2)

BETTER!Status = doc.ColumnValues(2)

10%Faster

Status is the third column of the view that was used to get to this document

Status is the third column of the view that was used to get to this document


Looping

WORKSDo

z=z+1y=y+1

Loop While y

Put the Smaller Bounds on the Outer Loop

WORKSfor q=0 to 5000for y=0 to 2

z=z+x(q,y)next

next

BETTER!for y=0 to 2for q=0 to 5000

z=z+x(q,y)next

next

BETTER!BETTER!for y=0 to 2for y=0 to 2for q=0 to 5000for q=0 to 5000

z=z=z+x(q,yz+x(q,y))nextnext

nextnext

400%Faster

LotusScriptLotusScript

Branching with IF Statements

WORKSif a=x and totalPrice()>0

(has to compute totalPriceeven if a x)

BETTER!if a=x then if totalPrice()>0

(only computes totalPrice if a=x)

BETTER!if a=x then if totalPrice()>0

(only computes totalPrice if a=x)

LotusScriptLotusScript

When Doc is Open in the Notes UI

WORKS BETTER!

Updates ui doc only once, after all fields are written

BETTER!

Updates ui doc only once, after all fields are writtenLotusScriptLotusScript

5xFaster

Demonstration: Autoreload Property

Profile Docs for Config Data

Profile Documents are cached Built-in programmatic access: Formula: GetProfileField() LotusScript/Java: NotesDatabase.GetProfileDocument()

But how do you manually access profile docs? End users: Profile Docs (open source) form from Andre

Guirards blog at www.ibm.comf Shows profile docs in the parent db containing the form

Geeks: NoteMan Toolbar (free) shows view of profile docs f Shows profile docs in any database

Demonstration: Profile document management

(See Designer Help Properties that improve database performance)

Miscellaneous Database Properties

Shared Elements in Notes Client

Shared fields/actions, subforms, script libraries stored in separate notes

Domino must make one call to get the document, one call to get the form, and one call for each shared element 10 shared fields = 10 extra design element fetches More than three computed subforms can have an exponential

effect on performance Shared elements are great for design maintenance but

use with caution in Notes Client!

Tradeoff

Shared Elements on the Web

In web applications, requesting and assembling of shared elements is done right on the server, without network activity No real performance impact go for it!

Quiz

Question: Whats the biggest Domino performance problem?

Answer: View indexing

What Well Cover


View Indexing

The top performance killer in Domino isView Indexing Every time a document is saved, Domino updates the

indexes of all auto-indexed views which select the document Including Notes form, web submit, and agent updates to a

document Indexing performance is affected by

f Number of documents in viewsf Number of fieldsf Frequency of updatef Number of viewsf Complexity of views

View Indexing tactics

Use a separate physical disk for view indexing aid (VIEW_REBUILD_DIR notes.ini variable)

Delete unused viewsf Determine which views are really being used

How to know which views are being used?

Use the Admin Clienta. Purge suspect indexesb. Allow user activity to resume for a week/monthc. Later, look for re-created indexes (they are active views)d. Remove unused views (Size=0) from design

Views: Time @Functions

Avoid @Today, @Now in view formulas Forces the calculation to be done every time the view

is opened H o w m a n y s e c o n d s d o e s it ta k e to In d e x a V ie w ?

T he se d a ta ba se s co n ta in o n ly 1 V ie w a nd 1 F o rmT he V ie w ha s o n ly o ne co lum n , a nd the S e le c tio n F o rm u la is @ A ll

1 0 1 0 0 1 0 0 00

5

1 0

1 5

2 0

2 5

3 0

3 5

N u m b e r o f D o c u m e n ts

S e c o n d s

F ie ld N a m e@ To d a y@ N o w

Views: Time @Functions (continued)

To show a view of documents more than N days old: A simple daily scheduled agent moves docs in/out of the old

documents folder (do not use a view)f Putting a document in a folder does not modify it --

minimizes replication activity and conflicts Or, daily agent modifies hard-coded date in view formula

f NotesView.SelectionFormula Or, reduce view index refresh interval

View Complexity

A p p ly in g D iffe re n t A ttr ib u te s to V ie w C o lu m n sN u m b e r o f M in u te s to B u ild a V ie w w ith 5 C o lu m n s

1 0 0 F ie ld s1 0 0 ,00 0 D o cu m e n ts

0

5

1 0

1 5

2 0

2 5M in u te s

P L A INF O R M U L A S (5 )S O R T (1 )S O R T (4 )C AT E G O R IZ E (1 )C AT E G O R IZ E (4 )

Views: User Sortable Columns

Count as viewsEach sort option has its own view index

WarningWarning

Views: Readers Restrictions

Readers restrictions can cause slow view display if ... User has Read access to only a small portion of many docs Domino must scan until it finds a pages worth (usually 30) of

accessible docs, or all the way until the end of the view!

WarningWarning

What Well Cover


Should I use WebQueryOpen?

R6, R7 and R8 have shown tremendous improvement regarding Open Agent performance. The horror stories have come from pre R6 days. Use R7 Agent Profiler to probe these agents.fMore on this later

WebQueryOpen LotusScript agents not quite as fast as, but offer more flexible programming functionality than, formulas.

WindowOnload Problem

Developer uses the onload event to start Web apps on page loads The problem: Onload does not fire until all page content has

been loaded, including images If you have any scripts that depend only on the DOM to be

loaded, then use a domReady.js as the kick offf Get your copy of js_domready.js on

DominoPerformance.com thanks to Dwight Wilbanks

function FormOnLoad( ) {~ your processes }

var FormLoader = new domFunction( FormOnLoad,"")

Use Cascading Style Sheets (CSS)

Separates your style and data Smaller HTML pages = faster loading CSS, JS files are cached in the browser Make your CSS efficient and standardized

Domino and Doctypes

By default domino throws out an incomplete doctype 6.5.3 notes.ini parameter sets the doc type server wide:fDominoCompleteDoctype=n 0 = Domino default 1 = Standard 2 = Strict R7 - a field $$HTMLFrontMatter sets the doctype per page

Why Doctypes are important

With the default or incomplete Doctype the page runs in Quirks Mode This has extra parsing for older HTML This slows down the document parsing which slows the

page display The right choice is to run in Standards Mode. This expects well formed HTML and XHTML This displays the page fasterfNot noticed on the server, but faster appearance to

end user

Use Server Web Site Rule to maximize file caching

Drastically improve your users Web Cache The users browser will cache many files, but it still needs

to check to see if it has the current version each timefThis can cause 2-3 extra seconds to each page load.

To fix this we need to tell the users browser that certain files are good for N days

Admin needs to set it but Developer needs to code it

Add Web Site Rule

Items in:

Domain//cachePath/

Will be cached

If-Modified-Since header variable

GZip Will Improve Performance

GZip (GNU Zip) is a common compression found on many Web Servers such as Apache, IIS,..

First discovered on Domino in R6.5x but only in DWA There have been many attempts to use this built in

GZip functionality but most have been reported as highly unstable.

Which brings us to

Use Web Site Rules to use GZip

Because browsers know about GZip, we can feed them GZip files We can do this for any external files (CSS, JS, etc.) To do this, make a web site rule to handle these items

and place them in a gz directory path How to use GZip

1. Make the Web Site Rule2. Compress your files in GZip format (www.gnu.org)3. Add these files in shared resources with a gz/ prefix4. Reference these files when needed

Add Web Site Rule for GZip

Items in:

Domain//gz/

Will be served as GZip

What Well Cover


Server Document: Maximum Concurrent Agents

For agent-intensive applications Allow more than one agent to run at the same time

Monitor your resource utilization after such a change to ensure your system has the CPU and memory to deal with increased activity

Server Document: Web Maximum Concurrent Agents

Allow more than one agent to run at the same time on the Web

Enabled To allow more than one agent to run on the Web server at the same time (asynchronously) Disabled (default) To run only one agent at a time (serially)

NOTES.INI Application Performance Variables

VIEW_REBUILD_DIR= Use a separate physical disk for view indexing

f Greatly improves overall performance on index-heavy servers

Updaters = [number] Run multiple update tasks to keep view indexes updated Set [number] = number of processors

NSF DbCache_Maxentries = [number] Number of databases that can be cached at one time Default is 25 or NSF_Buffer_Pool_Size divided by 300 KB If ratio of Database.DbCache.Hits to InitialDbOpen is low, then

consider increasing [number] Console command Dbcache disable closes cached databases

Monitoring Transactions vs. Users

Server_Show_Performance=1 in notes.ini Post to the server console every 60 seconds Number of transactions the server performs per minute and the

current number of active users on the system Helpful in tracking system performance over a period of time Transactions include router, agent manager, replication, virus

scanner, and user actions

[01A8:0021-0254] 04/10/2008 04:30:47 PM 6 Transactions/Minute, 0 Notes Users[01A8:0021-0254] 04/10/2008 04:31:47 PM 0 Transactions/Minute, 0 Notes Users[01A8:0021-0254] 04/10/2008 04:32:47 PM 4 Transactions/Minute, 0 Notes Users

Configuring Server Settings

Administrators own these settings, but developers must be aware of them

Both parties need to collaborate to determine best settings for each server

Make sure to monitor resource utilization before and after making such changes so that you can determine if you have enough hardware to benefit (and not make it worse!)

What Well Cover


Performance Debugging: Users perspective indications

Large, continuous tables on formsExcessive DbLookup/DbColumn operationsInefficient WebQueryOpen/WebQueryCloseLotusScript events

Creating/opening/saving document

Time functions in view formulasLarge active views with broad selection formulasExcessive view categorization

Saving a document (submitting from a browser, agent-based saves of other docs)

Time functions in view formulasReaders restrictions on documentsIndex property not set to Automatic

Opening a view

Possible ProblemPerformance Symptom

DDM (R7 and later)

Domino Domain Monitor contains super agent statistical tools What agent is causing problems? Time to run, Memory used

Wrapped in a Domino DB for easy reporting

Agent Profiler (R7 and later)

Profile an Agent to get detailed performance information Step 1 mark the agent to be profiled Step 2 Run the agent Step 3 View

Agent Profiler Sample Output

Fiddler Example Output

Undocumented Notes Feature: Watching NRPCs

Watch behind the scenes querying of forms, subforms, views (not just code) Remote Procedure Calls (RPCs) = Notes-to-Domino talk See http://MartinScott.com Resources Technical Articles

Notes Client Domino Server

Demo: Watching Notes RPCs

Undocumented Notes Feature: Uncover Performance Bottlenecks

Enable RPC watching on Notes client NOTES.INI file on Notes client/Designer

f Client_Clock=1f Debug_Console=1f Debug_Outfile= (optional)

Restart Notes client Remove from NOTES.INI and restart to disable RPC watching

(1-81) OPEN_DB:(2-83) OPEN_COLLECTION: 429 ms. [32+626=658](3-83) READ_ENTRIES: 8354 ms. [46+64432=64478](4-91) CLOSE_COLLECTION: 0 ms. [12+0=12](5-92) DB_REPLINFO_GET: 438 ms. [14+32=46](6-92) DB_REPLINFO_GET: 169 ms. [14+32=46]...

Watching Notes RPCs Opening my mailWatching Notes RPCs Opening my mail

Understanding RPCs

Common RPCs and What They Indicate

(20-121) OPEN_NOTE: 1045 ms. [28+3906=3934]

(Seq. #), RPC_NAME, time, [bytes_sent+bytes_received=total]

RPC_Name DescriptionOpen_Session Authenticate with the server and establish a sessionOpen_Database Find and open a database

Open_Note Get the contents of a note (data document, design element, or ACL info)

Update_Note Save a document Open_Collection Open a viewRead_Entries Get a list of document information from a view or search

Find_By_Key Find a document via DBLookup or LotusScript GetDocumentByKey

Get_Special_Note_ID Get ACL informationClose_dB Close database session

(1-14 [1]) OPEN_DB(CN=Vienna01/O=MSC!!mail\jmagee2.nsf): (Connect to Vienna01/MSC: 0 ms) (Exchnames: 0 ms) (OPEN_SESSION: 561 ms)(2-16 [3]) GET_UNREAD_NOTE_TABLE: 571 ms. [134+290=424](2-17 [4]) DBGETREPLICAMATCHES: 1692 ms. [290+17452=17742](3-17 [5]) OPEN_NOTE(REP85257246:005DD81C-NTFFFF0010,03000400): 841 ms. [176+780=956](3-18 [6]) OPEN_DB(CN=Vienna01/O=MSC!!mail\jmagee2.nsf): 621 ms. [48+1886=1934](4-18 [7]) GET_NAMED_OBJECT_ID($profile_015calendarprofile_): 641 ms. [134+290=424](4-18 [8]) READ REPLICATION HISTORY: 541 ms. [54+24=78](5-18 [9]) OPEN_NOTE(REP85257246:005DD81C-NT00002FE6,00400020): (6-19 [10]) POLL_DEL_SEQNUM: 571 ms. [60+64=124] (Cache entry not found)(5-19 [10]) DB_REPLINFO_GET: 821 ms. [14+32=46]1342 ms. [48+9050=9098](7-20 [11]) GET_NAMED_OBJECT_ID($profile_024archive database profile_): (6-20 [12]) SEARCH: (Connect to Vienna01/MSC: 1382 ms) 581 ms. [64+24=88](8-20 [13]) OPEN_NOTE(REP85257246:005DD81C-NT000010F2,00400020): (OPEN_SESSION: 531 ms)520 ms. [48+214=262](9-21 [15]) OPEN_COLLECTION(REP85257246:005DD81C-NT0000073E,0040,4008): 1292 ms. [70+782=852](7-21 [16]) OPEN_NOTE(REP85257246:005DD81C-NTFFFF0040,03000400): 551 ms. [110+28=138]1061 ms. [48+1534=1582]

Watching Notes RPCsWatching Notes RPCs

How to see what the NoteIDs are

Use a free tool like NoteMan to look up the NoteID00002FE6

This item is a Calendar Profile doc, and took .571 seconds of my DB open time.

(5-18 [9]) OPEN_NOTE(REP85257246:005DD81C-NT00002FE6,00400020): (6-19 [10]) POLL_DEL_SEQNUM: 571 ms.

NRPC Parser open source tool

Runs on NRPC output files to translate IDs and commandsf Much easier to find and understand problems

Developed by Jamie Magee and Andrew Magerman Download from www.JamieMagee.com or www.Magerman.com

What Well Cover


Resource Monitoring

Use operating system tools PerfMon (Windows)

f Start \ Control Panel \ Administrative Tools \ Performance PerfMeter (UNIX) PerfMon / PEX (AS/400)

Use during live use to see bottlenecks

Resource Monitoring

Example: agent is bound by CPU, more RAM would not helpLimited by Processor

Resource Monitoring (continued)

Example: View index rebuild showing that more RAM and faster drive would help, but more CPU would not.

Resource Monitoring: Identifying areas to improve

HighMedium

Increase server RAMOptimize .INI file memory settings

Memory

HighMedium

Increase network capacityTake advantage of browser caching

Network

HighHighMediumMediumMedium

Reduce view complexity, # and sizeOptimize code implementationOptimize server .INI file settingsDisable server screen saverTake advantage of browser caching

Processor

HighMediumLow

Reduce view complexity, # and size Optimize cache settings in .INI fileTake advantage of browser caching

Disk Access

Improvement Potential

Area of FocusLimiting Resource

What Well Cover


Resources References

Lotus developerWorks performance zone: www.ibm.com/developerworks/lotus/performance

f Index to all IBM performance articles www.redbooks.ibm.com Any Domino book with Performance in the name Several other good ones

MartinScott.com 30-page performance article: http://MartinScott.com Resources Technical Articles

NoteMan Toolbar www.NoteMan.com Andre Guirards blog: Best Practice Makes Perfect http://www-10.lotus.com/ldd/bpmpblog.nsf

Some ideas from Kevin Marshall and Andrew Magerman www.FastNSF.com Server performance tips

Key Points to Take Home

Beware of code and design gotchas! Refine and minimize view indexing, remove unused

views Employ web standards and take advantage of fast web

service techniques Developers and admins should collaborate on server

configuration settings DDM, Agent Profiling, and NRPC monitoring can tell you

a lot about your application code Resource monitoring reveals the pulse of the server,

and indicates hardware needs

Your Turn!

Questions?

How to contact me:JMagee at MartinScott.com

/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /Unknown

/Description >>> setdistillerparams> setpagedevice