page 1 © 2001 hewlett-packard company tools for measuring system and application performance...

© 2001 Hewlett-Packard Company

Tools for Measuring System

and

Application Performance • Introduction

• GlancePlus• Introduction

• Glance Motif

• Glance Character Mode

• Glance Command-line

• Xverbosegc

• HPjmeter and –Xeprof

• Other Tools

• gdb


GlancePlus Motif

/opt/perf/bin/gpm

Tools for Measuring

Performance


GlancePlus Motif

• Open multiple windows to simultaneously view:

• Machine’s behavior

• Process’s behavior

• Thread behavior

• We will learn how to use it with Java applications

Tools for Measuring

Performance


GlancePlus Motif Main Window – Overview of System

Tools for Measuring

Performance


GlancePlus MotifMain Window – Overview of System

CPU

I/O

Memory

Network

Tools for Measuring

Performance


GlancePlus MotifConfigure Sampling Time

Tools for Measuring

Performance


GlancePlus MotifConfigure Sampling Time

Configure:Measurement

Sample Interval:Minutes: 0Seconds: 1

Graph Points: 50

Tools for Measuring

Performance


GlancePlus MotifMain Window – CPU Usage

Active Button: CPU Graph

Tools for Measuring

Performance


GlancePlus MotifCPU Graph

Tools for Measuring

Performance


GlancePlus MotifCPU Graph

CPU Queue Length

8 CPUs

CPU Usage Categories:• Real• Negative Nice (-20 to 0)• Nice (greater than 0)• Normal User• System

Tools for Measuring

Performance


GlancePlus MotifCPU Graph – Garbage Collection

1 Thread Active during GC

Tools for Measuring

Performance


GlancePlus MotifMain Window - Reports

Tools for Measuring

Performance


GlancePlus MotifMain Window - Reports:Process List

Reports:Process List

Tools for Measuring

Performance


GlancePlus MotifProcess List

Tools for Measuring

Performance


GlancePlus MotifProcess List

8 CPU System: 800% Maximum CPU%

Tools for Measuring

Performance


GlancePlus MotifProcess List – Arrange Columns

Tools for Measuring

Performance



Configure:Arrange Columns

Tools for Measuring

Performance



Tools for Measuring

Performance



Interesting values:• CPU%• User CPU%• System CPU%• Virtual Memory• Res(ident) Memory• Physical I/O• Configure:Choose Metrics:• PROC_THREAD_COUNT

Tools for Measuring

Performance


GlancePlus MotifProcess List - Reports

Tools for Measuring

Performance



Process ResourcesProcess Open FilesProcess Memory RegionsProcess System CallsProcess Thread List

Tools for Measuring

Performance


GlancePlus MotifProcess System Calls

Tools for Measuring

Performance



Sorted on: SysCall Rate

Tools for Measuring

Performance


GlancePlus MotifProcess System Calls - Sort

Configure:Sort Fields

Tools for Measuring

Performance


GlancePlus MotifProcess System Calls - Sort

Tools for Measuring

Performance



System Calls for Idling JVM

Tools for Measuring

Performance




Tools for Measuring

Performance


GlancePlus MotifProcess Memory Regions

Tools for Measuring

Performance



VSS – Virtual Set Size

Total Reserved Space

RSS – Resident Set Size

Space Actively in Use

Total Number of Memory Regions

Tools for Measuring

Performance



Data == C Heap

Text == Executable

Other == Java Heap Shared Libraries Thread Stacks

Stack == For 1st Thread

Tools for Measuring

Performance


GlancePlus MotifProcess Memory Regions - Sort

Tools for Measuring

Performance


GlancePlus MotifProcess Memory Regions - Sort

Sort on VSS

Tools for Measuring

Performance


GlancePlus MotifProcess Memory Regions - Metrics

Add Page Size Metrics

Tools for Measuring

Performance



Tools for Measuring

Performance



Select All

PROC_REGION_PAGE_COUNT*

Tools for Measuring

Performance



For Address: Select

PROC_REGION_VIRT_ADDRS

Tools for Measuring

Performance


GlancePlus MotifProcess Memory Regions – Pages

Page Sizes AllocatedIn Java Heap

All: 4 KB

Too ManyTotal!!

Tools for Measuring

Performance


GlancePlus MotifProcess Memory Regions – Pages

Page Sizes AllocatedIn Java Heap

All: 4 MB

Very FewTotal!!

Tools for Measuring

Performance




Tools for Measuring

Performance


GlancePlus MotifProcess Thread List

Tools for Measuring

Performance


GlancePlus MotifProcess Thread List

Active Threads

JVM Threads

Total Number of Threads

Thread Id

Tools for Measuring

Performance




Tools for Measuring

Performance


GlancePlus MotifProcess Open Files

Tools for Measuring

Performance


GlancePlus MotifProcess Open Files

File Type

Sockets – Monitor Bytes Read and Written!

File Name

File Offset

Total Number of Open Files

Tools for Measuring

Performance


GlancePlus MotifMain Window - Reports

System Info: System Tables Graph

Tools for Measuring

Performance


GlancePlus MotifSystem Tables Graph

Process TableFile TableShared Memory TableSemaphore TableFile LocksSwap Space

Tools for Measuring

Performance


GlancePlus MotifIdentifying Performance Problems

• Patterns to recognize• High System CPU time

– Often associated with monitor contention

– Check by looking at System Calls– High sched_yield, ksleep, kwakeup call rate confirms

• Increasing memory usage– Thread stacks

– C Heap

– Percentage of the Java Heap in active use

Tools for Measuring

Performance


HotSpot JVMObject Model

• Efficient and low overhead

header

non-staticfields

methodtable ptr

staticfields

GC maps

header

C++ vtable

Class

Object

Tools for Measuring

Performance


HotSpot JVMObject Header

• Age (7 bits)

• Hash (23 bits)

• Lock (2 bits)• lock

• unlock

• pthread mutex used (inflated)

• used by mark/sweep

LockHashAge

Tools for Measuring

Performance


HotSpot JVMEfficient, Low-Overhead Locking

Stack

header

non-staticfields Object

ULheaderL

If another thread tries to obtain the same

monitor: Lock inflation (uses a pthread mutex)

Tools for Measuring

Performance


M onitor/M utex

thread 1 thread 3

M onitor/M utex

thread 1 thread 2thread 3

M utexLock

M onitor/M utex

thread 1 thread 3

M onitor/M utex

thread 1thread 3

M utexLock

M onitor/M utex

thread 1

How Does a pthread mutex Work?

• Try to get lock immediately

• If fail, be optimistic:– Enter queue and wait

– Repeat: spin, sched_yield

– Next waiter given lock

• Until: not optimistic– ksleep

Has 2 WaitersHas 2 Waiters

Tools for Measuring

Performance


Contention: System Impact

• Monitors • Implemented using a single word in memory that serves as

the “lock word”

• On a system with multiple CPUs:• All of the caches on the individual CPUs must be updated

during the spin-sched_yield() cycle

• Consequences:• Threads time is spent accessing one word – spinning

– Result: High CPU usage with little application work

• Machine spends all of its time maintaining cache coherency!!

• More threads yield lower performance!

Tools for Measuring

Performance


M onitor/M utex

thread 1 thread 3

M onitor/M utex

thread 1 thread 2thread 3

M utexLock

M onitor/M utex

thread 1 thread 3

M onitor/M utex

thread 1thread 3

M utexLock

M onitor/M utex

thread 1

Contention: System Impact

CPU 1

Cache

CPU 2

Cache

CPU 3

Cache

CPU 4

Cache

Memory

thread 1

thread 3

thread 4

thread 5

thread 7

thread 6

thread 8

Cache Coherency

Cache CoherencyLDCW

Tools for Measuring

Performance


GlancePlus MotifExample of High Contention

Tools for Measuring

Performance



HIGH SYSTEM CPU TIME

LOW USER CPU TIME

Tools for Measuring

Performance



Tools for Measuring

Performance



HIGH sched_yield() ksleep() kwakeup()CALL RATES

Tools for Measuring

Performance



User

System

Tools for Measuring

Performance


GlancePlus MotifExample of High Contention - Fixed

Tools for Measuring

Performance



Tools for Measuring

Performance


GlancePlus MotifExample of High Contention - Fixed

2 Fold Improvement!

5 Fold Improvement!

100x Reduction!

Tools for Measuring

Performance


GlancePlus MotifMonitoring Memory Usage

• Select process ID for process in Process window and select Report:Memory Regions

• Data VSS (Virtual Set Size - allocated) represents the C heap • Text VSS represents memory for the executable• Other VSS represents the memory used by shared libraries,

thread stacks, Java Heap, Code cache, …• Private VSS (part of Other) contains the Java Heap and

Thread stacks

• Corresponding RSS (Resident Set Size) for each area can also be monitored

• Watch for significant growth in any of these regions• Use the Glance adviser syntax for long term

monitoring

Tools for Measuring

Performance


GlancePlus MotifConclusions

• Motif version is a powerful tool

• Exposes problems with Java performance

• Easy to do systematic analysis using displayed information

page 1 © 2001 hewlett-packard company tools for measuring system and application performance...

Documents

hewlettpackard company

hewlettpackard company

measuring performance

tools gdb slide

measuring system

gc slide

count slide

cpu system