2006.02.01 wonyoung lee for web based system performance analysis mathematical approach on

2006.02.01http://www.javaservice.com

WonYoung Lee

for Web based System

Performance Analysis Mathematical Approach on

2

Question: What is the definition of Performance?

A.jsp B.jsp

http://192.168.0.2/a.jsp http://192.168.0.2/b.jsp

Unit Response time

0.5 secondsUnit Response time

4.0 seconds

3

Application A: Response Time Grahp

5 10 15 20 25

5

10

Concurrent Users

Resp

on

se T

ime(s

ec)

1

A.jsp B.jsp


Unit Response Time

0.5 secondsUnit Response Time

4.0 seconds

4

5 10 15 20 25

5

10

Concurrent Users

Resp

on

se T

ime(s

ec)

A.jsp B.jsp


Unit Response Time0.5 seconds

Unit Response Time

4.0 seconds

Application B: Response Time Graph

5

Comparison of A and B

5 10 15 20 25

5

10

Concurrent Users

Resp

on

se T

ime(s

ec) A

B

6

1. Objective of Performance

1.How many Clients2.Reasonable response time3.How to test or measure

7

1.1 Visit time & Think time 1. Definition

Click ! Click !

Response Time Think Time

Request Interval

Request Interval(sec) = Response Time(sec) + Think Time(sec)

Visit Time

New User Visited

Click ! Click ! Click ! Click ! Click ! Click !

Leave

8

1.2 Think time

Click ! Click !

Response Time Think Time

Request Interval

ThinkTime look like a constant value for the specific business domain

1. Definition

9

1.3 Concurrent User

Concurrent Users : 6 Time

Request Interval( ≈Think Time)

D

iffere

nt

IP A

dd

r ess

es

Visit Time

Definition

Concurrent User = ActiveUser + Inactive User

Note: HTTP is connection-less

1. Definition

Max Thinktime

Note: How many active users?

10

Request Interval

Response Time

Think Time

Concurrent User

1.4 Active UserTim

e

Click

Click

Active User

Active Service

Worker-Thread

System

1. Definition

11

1.5 Throughput

Time

Throughput(tps) =number of request

measure time(sec)

Unit : tph, tpm, tps, (pps, rps, ops, hit/sec)3,600 tph = 60 tpm = 1 tps

Measure Time

Definition

1. Definition

NOTE: Arrival Rate, Service Rate

12

2. Request/Response System Model

1.Mathematical Approach2.Queuing Theory3.Quantitative Analysis4.Measuring

13

2.1 Request and Response2. Request/Response System Model

Average Response Time

Throughput

Assumption: No ThinkTime

Question:What is the ability of her ?

Right Answer:Maximum throughput

# of Test Users

# of Test Users#

per

min

ute

s

Ave. R

esp

. Tim

e

14

2.2 MeasuringThroughput2. Request/Response System Model

Response Time

Virtual User (Assumption: ThinkTime=0)

Incr

easi

ng

Virtual User(ThinkTime=0)

Throughput(tps)

ResponseTime(sec)

15

2.3 Little’s Law 2. Request/Response System Model

Response Time(sec)

ActiveUser(ThinkTime=0)

Number of ActiveUser = Throughput(tps) x Average Response Time(sec)

Number of ActiveUser

Average Response Time(sec)Throughput(tps) =

ActiveUser

Throughput(tps)

ResponseTime(sec)

N

R

T

N = T x R

16

Active Service Request Interval

Response Time

Think Time

Concurrent User

Worker-Thread

2.4 ActiveUser’s Law

System Click

Ave. ResponseTime(sec)

Ave. ResponseTime(sec) + ThinkTime(sec)

ActiveUser = ConcurrentUser x


Active User

ResponseTime’s Law

17

2.4.1 Proff of ActiveUser’s Law2. Request/Response System Model

ActiveUser

Resp.Time(sec)Throughput(tps) =

ConcurrentUserRequest Interval(=Resp.Time+ThinkTime)

Throughput(tps) =

Little’s Law

Resp.Time(sec)

Request Interval(=Resp.Time+ThinkTime)ActiveUser = ConcurrentUser x

ActiveUser

Resp.Time(sec)ConcurrentUser

Request Interval(=Resp.Time+ThinkTime)=

18

2.4.2 Meaning of Active User -1-

Request Rate Service Rate

Active User

Ave. ResponseTime(sec)

Ave. ResponseTime(sec) + ThinkTime(sec)

ActiveUser = ConcurrentUser x


19

2.4.3 Meaning of Active User -2-2. Request/Response System Model

20

2.5 Concurrent User Equation2. Request/Response System Model

Definition

Equations

ResponseTime

Resp.Time + ThinkTime

= ActiveUser x ( 1 +

ConcurrentUser = ActiveUser x

Resp.Time(sec)

Request Interval(=Resp.Time+ThinkTime)ActiveUser = ConcurrentUser x

= Throughtput x ( ResponseTime + ThinkTime)

ActiveUser

Resp.Time(sec)Throughput(tps) = Little’s Law

)

= ActiveUser + ( Throughput x ThinkTime)

①

②

③

(Note: Throughput(tps) : ArrivalRate or ServiceRate)

ResponseTimeThinkTime

21

-Permanent Cookie-Thinktime-Arrival Rate-Responsetime-Active User(Service)

-- Concurrent User

2.6 Example of Concurrent User Monitoring2. Request/Response System Model

22

2.7 SLA & ThinkTime

SLA(Service Level Agreement) - Concurrent User : 5,000 - Average Response Time: less than 3 sec

SLA(Service Level Agreement) - Concurrent User : 5,000 - Think Time = 30 (Example) - Average Response Time: less than 3 sec

ThinkTime Agreement- Tele-market : 10-15sec- MIS Intranet : 15-20 sec- Internet Banking : 25-35 sec- Online Shopping Mall : 30-40 sec- Community : more longer Real ThinkTime Data

+


23

2.8 Performance Test ThinkTime = 30

Virtual User(ThinkTime=30 ConcurrentUser)

Thro

ughput(

tps

)

5,000

Resp

onse

Tim

e

1.5

235.3

8,000

4.0

Throughput Graph

AverageResponseTime Graph158.7

ConcurrentUser = Throughtput x { ResponseTime + ThinkTime(30) }

5000 = 158.7 x ( 1.5 + 30 ) 8000 = 235.3 x ( 4.0 + 30 )


24

2.9 Performance Test ThinkTime=0

Virtual User(ThinkTime=0 ActiveUser)

Thro

ughput(

tps

)

238

Resp

onse

Tim

e

158.7

235.3

941

Throughput Graph

Average Response Time Graph

1.5

4.0

ConcurrentUser 5,000 8,000

ConcurrentUser = ActiveUser + Throughput(tps) x ThinkTime(sec)

ConcurrentUser = ActiveUser x { 1 + ThinkTime(sec)

Resp.Time(sec)}

ConcurrentUser = Throughput(tps) x { Resp.Time(sec)+ThinkTime(sec)}


25

2.10 Queuing Theory - G/G/1

λ μ

Response TimeUtilizationActiveUser

Throughput (λ<μ)

(λ<μ)

Arrival Rate Max Throughput


26

Concurrent User

req/s

ec

??

Concurrent User Concurrent User

Arrival Rate(λ)

Active User(N) Response Time(R)

Response TimeActiveUser

Arrival rateArrival rate

2.11 Maximum Concurrent User2. Request/Response System Model

27

2.12 Saturation Point, Buckle Zone2. Request/Response System Model

28

Tuning ? What’s mean?

2.13 Understanding of Throughput Graph2. Request/Response System Model

29

2.14 Throughput and Active User

Demo


30

3. Multiple Applications

- Different Hit Ratio- Different Performance- Homogeneous/heterogeneous Bottleneck Condition


31

3.1 2-Application model3. Multiple Applications

40

TPSmax

20

40

20

A B

40

20

10

40

30

20

A BA B 5

40

1015

A B

Active Threads

40TPS

Saturation Point

A application

40

20

TPS

Saturation Point

B application

Active Threads40

Active Threads

40TPS

4020

20

10

A

B

Active Threads

40TPS

4010

30

5

A

B

30 Active Threads

40TPS

40

1510

B

A

10 30

Graph 1 Graph 2 Graph 3

1 : 1 3 : 1 1 : 3

TPSmax TPSmax TPSmax

32

3.1.1 2-Applicatoin Model

40A

20

10

20

30

B10

graph 1

graph 3

graph 2

O

Different Arrival Rate

Limited by same resource bottleneck

Homogeneous bottleneck condition


33

xr1 r2

3.1.2 2-Applicatoin Model

T2

y

T1

x

xT1

yT2

+ = 1

=

Q

P

O

(r1,r2)r2

r1

A: r1 (req/sec)B: r2 (req/sec)

R2

R1

y


34

3.2 3-Application Model

Q

O T2

T3

y

z

P

(0, T2, 0)

(0, 0, T3)

(0, 0)

r1

r2

x

T1

r3

x

T1

y

T2

+ = 1z

T3

+

xr1

yr2

=zr3

=

(T1, 0, 0)

(r1, r2, r3)


35

3.3 n-Applicatoin Model

NOTE: Under Homogeneous Bottleneck Condition


( Critical Inequality Performance Equation )

36

3.4 Performance Utilization

T2

T3

y

z

x

T1

x

T1

y

T2

+ = 1z

T3

+

App. Ti

TPSmax

ri

req/sec

ri / Ti

x 400 48.2 0.12

y 9 1.2 0.13

z 50 22.7 0.45

0.12 + 0.13 + 0.45 = 0.70

∑ { ri / Ti } x 100 = 70 %

Q

P

∑ { ri / Ti } ≤ 1.0


Utilization of Critical Performance

37

3.5 Revision of Utilization of Critical Performance

App Ti

TPSmax

ri

req/sec

ri / Ti

a1 42 5.4 0.129a2 9 1.9 0.210a3 70 10.7 0.153a4 15 4.5 0.300

Sub ∑ri =22.5 ∑ { ri / Ti } = 0.79

… … … …

TOTAL 25(req/s)

a.jspa.jsp : 5.4 req/sec

b.jspb.jsp : 1.9 req/sec

c.jspc.jsp : 10.7 req/sec

d.jspd.jsp : 4.5 req/sec

…..

Arrival RateArrival Rate

∑ { ri / Ti } ≤ ρ* 1.0 (ρ is total hit ratio of ∀ai)

∑ { ri / Ti } / ρ = ∑ { ri / Ti } / (22.5/ 25) = 88%


38

Q

O

z

P

(0, 0)

r2

x

r3

xr1

yr2

=zr3

=

(r1, r2, r3)

Arrival rate line

bottleneck saturation flat

Bottleneck surface

∑ { ri / Ti } ≤ 1.0 +ε(ε>0)

3.6 Multiple Bottleneck Theory

Heterogeneous bottleneck condition


Critical Inequality Performance Equation (final)

39

3.7 Performance Matrix

App. Arrival Rate(tps) Arrival Ratio MaxThroughput

λi /Τi

a1 λ1 λ1 /λ=ρ1 Τ1 λ1/Τ1

a2 λ2 λ2 /λ=ρ2 Τ2 λ2/Τ2

a3 λ3 λ3 /λ=ρ3 Τ3 λ3/Τ3

… … … …

an λn λn /λ=ρn Τn λn/Τn

(Others) λothers 1-ρ(Ex.20%) N/A N/A

SUM ∑ λi + λothers = λ ∑ρi + 0.2 = 1.0 N/A ∑ λi/Τi

∑ λi/Τi ≤ ρ(1.0+ε) (ε≥0)

(ρ is total hit ratio of a∀ i)Critical Performance Utilization:

∑ λi/Τi

ρ(1.0+ε)(When ConcurrentUser = N)(ε≈ 0)

≤1.0

NOTE: 8:2 Rule


40

3.8 Objective of Performance Estimation

Arrival RateConcurrent Users

N λ

kN kλ

Current Performance Utilization :

∑ λi/Τi

ρ(1.0+ε)1.0 (100%)

x k

Current Current

FutureFuture


How many concurrent users can be accepted?

How many times than now?

41

3.9 Performance Test for Multiple Scenario

App. Ratioρi

a1 0.5

a2 0.2

a3 0.1

… …

(Others) 0.2(20%)

SUM 0.8(80%)

ThinkTime = 28.5 sec(Example)

Response Time Graph

Virtual User(ThinkTime = 28.5 sec)

942

Acceptable Response Time


42

테스트 명 Hit Count Arrival Rate R(Ratio) T(TPS)예상

Rmax(Req/ sec)

ArrivalRate/ T

R/ T 비고

1. (listItemInfo)공매공고 297 0.083 0.14 16.25 0.14Rmax 0.005 0.0086

2. (listitem)매각물건 288 0.080 0.13 1.5 0.13Rmax 0.053 0.0867 J AVA EXCEPTION ERROR

3. (viewitm_real)매각물건 180 0.050 0.08 18.5 0.08Rmax 0.003 0.0043

4. (index)홈페이지 116 0.032 0.05 2.75 0.05Rmax 0.012 0.0182 CPU 100% FULL

5. (searchItemByDetailInfo)파워검색 109 0.030 0.05 8 0.05Rmax 0.004 0.0063 8 DB 95% FULL명일때 서버6. (listImpendingBidItem)입찰임박물건 161 0.045 0.07 28 0.07Rmax 0.002 0.0025 CPU 70%웹 서버 사용율7. (searchZipAddress_01)주소검색 55 0.015 0.03 60 0.03Rmax 0.000 0.0005 CPU 100%웹 서버 사용율8. viewItem_bond물건정보 53 0.015 0.02 19.5 0.02Rmax 0.001 0.0010 CPU 90%웹 서버 사용율9. (listKamcoAucNotice)공매공고 48 0.013 0.02 2.5 0.02Rmax 0.005 0.0080 8 DB CPU 100%명일때 서버 사용율10. newAucNotice새로운공고 45 0.013 0.02 4 0.02Rmax 0.003 0.0050 6 DB CPU 100%명일때 서버 사용율11.FAQ ViewFAQ보기 44 0.012 0.02 24 0.02Rmax 0.001 0.0008 WEB CPU 80%서버 사용율12.LISTFAQListFAQ 43 0.012 0.02 19.5 0.02Rmax 0.001 0.0010 DB , WEB CPU 90%서버 서버 사용율13. -viewAucNotice새로운공고 43 0.012 0.02 24 0.02Rmax 0.000 0.0008 - FULL웹 서버 다운 세션14.ID checkDuplicateUserId중복확인 33 0.009 0.02 67 0.02Rmax 0.000 0.0003 CPU - 94%웹 서버 사용율15. listItemInfoBySched공매일정 60 0.017 0.01 22.5 0.01Rmax 0.001 0.0004 CPU - 90%웹 서버 사용율16. listAucNotice통합공고 26 0.007 0.01 9 0.01Rmax 0.001 0.0011 DB CPU 100%서버 사용율17. welcomeRegisterForm회원가입 24 0.007 0.01 22.5 0.01Rmax 0.000 0.0004 CPU 90%웹 서버 사용율18. createUserForm회원가입동의 22 0.006 0.01 23.5 0.01Rmax 0.000 0.0004 CPU 80%웹 서버 사용율19. listDocForm부가정보 21 0.006 0.01 32 0.01Rmax 0.000 0.0003 CPU 90%웹 서버 사용율20. searchItem파워검색 21 0.006 0.01 19 0.01Rmax 0.000 0.0005 CPU 90%웹 서버 사용율21. listNewItem새로운물건 19 0.005 0.01 4 0.01Rmax 0.001 0.0025 DB CPU 100%서버 사용율22. loginForm로그인폼 19 0.005 0.01 25.5 0.01Rmax 0.000 0.0004 CPU 90%웹 서버 사용율23. contractPersonalUser이용약관 17 0.005 0.01 24 0.01Rmax 0.000 0.0004 CPU - 70%웹 서버 사용율24. login로그인 16 0.004 0.01 3.5 0.01Rmax 0.001 0.0029 DB CPU 100%서버 사용율

Sum 0.489 0.79 0.79Rmax 0.095 0.1535

Maximum adaptable concurrent

user: 219

Expected maximum TPS : 6.515(tps)

Maximum adaptable concurrent user : 202

Expected maximum TPS : 6.719 (tps)

Result of Test using Multiple

Scenario at the same time

Result of Test using each application

test.

3.10 Example of Performance Test

1 + 1 = 2.5 ?


43

3.11 Termination about Performance Test

Load Test

Stress Test

Availability Test

Performance Test

44

Relative Performance Problem SQL Query bottleneck (DB index, Full Scan, Heavy Query)

Bottleneck on back-end transactions (CICS,TUXEDO, TCP/IP

Socket)

Relative bad performance on specific application

(synchronized, CPU time)

Relative bad performance on most of application caused by

too small H/W

Other side issue, for example, Network bottleneck.

Two Types of Performance Problem

Conditional Performance Problem JDBC Connection resource leakage

Memory leakage (need too large memory, Memory Leak, Native

Memory Leak)

Unbalanced WAS tuning (Pool Size, Number of Thread, Heap Size)

Caused by a bug on JVM/WAS/JDBC (Sybase JDBC, JVM Bug, WAS

Bug)

Thread Lock/Dead Lock (Application issue, Firewall issue)

45

Relative Performance Problem

Request Rate Service Rate

Active User

46

Relative Performance Problem: Ramp up Test

47

Relative Performance Problem: Ramp up Test

48

JDBC Connection resource leakage

Memory leakage (need too large memory, Memory Leak, Native Memory

Leak)

Unbalanced WAS tuning (Pool Size, Thread 개수 , Heap Size)

Caused by a bug on JVM/WAS/JDBC (Sybase JDBC, JVM Bug, WAS Bug)

Thread Lock/Dead Lock (Application/Framework issue, Firewall issue)

Database lock caused by uncommited nor unrollbacked

database issue (buffer full, unexpected Batch Job, ..)

upload or download a large file

unexpected infinite loop on an application: CPU 100%

Disk/memory Full

bad performance on specific application or specific usres

Conditional Performance Problem

49

S80(12-way,8GB, 78,126 tpmC)S80(6-way,6GB, 41,140 tpmC)

Visitors a day: 4,506 usersPeak Concurrent users: 275 usersPeak Arrival Rate : 18.3 tpsRequest Interval : 18 secVistors vs Concurrent users: 6.1%

CPU : ?(unkown, no issue)119,266tpmC/18.3tps = 6,517 tpmC/tps

3.1 Issue on Capacity Planning

3.1.1 K-university KMS System

50

3.1.2 B company

M80 (4-way, 3GB, 34,588 tpmC) x 2S80 (12-way, 8GB, 78,126 tpmC) x 26F1 (4-way, 4GB, 44,500 tpmC) x 1H80 (2-way, 1GB) x 8

[Seoul:M80(4-way)x2: 69,176 tpmC ]Visitors a day: 1,621(total 3,435)Peak Concurrent Users: 600 usersHits a day : 466,639 hitsPeak Arrival Rate : 20.0 tpsRequest Interval : 30 secAverage Visit time: 1:25:43 Visitors vs Concurrent Users: 31%CPU Utilization: 70-100%69,176tpmC/20tps= 3,459 tpmC/tps

6F1(4-way,4GB, 44,500 tpmC ) Added.CPU Utilization: 60%113,676tpmC/20tps= 5,684 tpmC/tps

51

3.1.3 N-bank Internet banking

M80(4-way, 4GB, 34,588 tpmC) x 3

Visitors a day: 96,753 usersPeak Concurrent Users: 1,500-2,000 usersHits a day : 1,795,867 hitsPeak Arrival Rate : 58.3 tpsRequest Interval : 25.7 secVisit time: 6min 25secAverage hits per visit time: 18.6 clicksVisitors vs Concurrent Users: 1.6-2.0%

CPU Utilization: 70%103,764 tpmC/58.3tps= 1,780tpmC/tps

52

3.1.4 J-bank CRM System

S85(12-way, 32GB, 124,818tpmC) X 2+ HOST DB

Visitors a day: 5,028 usersPeak Concurrent Users: 250 usersHits a day : 264,060 hitsPeak Arrival Rate : 10 tpsRequest Interval : 25.3 secVisit time: 4min 10secAverage hits per visit time: 9.9 clicksVisitors vs Concurrent Users: 4.97 %

CPU Utilization: 30-40% 249,636tpmC / 10 tps = 24,964 tpmC/tps

53

3.1.5 D-insurance e-Hanaro System

WAS 6H1(4-Way 4 GB, 40,763 tpmC)DB H70(4-Way 4GB, 17,134 tpmC)+HOST CICS

Visitors a day: 2,800 user(resistered 3,300)Peak Concurrent Users: 350 userHits a day : 301,190 hitsPeak Arrival Rate : 33 tpsRequest Interval : 9 secVisit time: 16 minAverage Hits per visit time: 120.5 clicksVisitors vs Concurrent Users: 12.5 %

CPU Utilization: 50%40,763 tpmC / 33 tps= 1,235 tpmC/tps

54

3.1.6 D company

WAS H80(2-way, 2GB, 14,756(?) tpmC)DB S80(12-way, 8GB, 67,908tpmC)

Visitors: 258 usersPeak Concurrent Users: 45 userHits a day : 65,192 hitsPeak Arrival Rate : 4.2 tpsRequest Interval : 11-18 secVisitors vs Concurrent Users: 17.4 %

CPU Utilization: 50-60% (DB CPU: 35%)14,756 tpmC / 4.2 tps= 3,513 tpmC/tps

55

3.1.7 K-bank eCRM System

WAS H80 (2-way, 4GB, 14,756(?) tpmC) x 2DB M80(2way-4GB, 18,647(?) tpmC)

Visitors a day: 37,951 usersPeak Concurrent Users: 230 usersHits a day : 235,527 hitsPeak Arrival Rate : 5.83 tpsRequest Interval : 24.7 secVisit time: 2 min 33 secAverage hits per visit time: 6.2 clicksVisitors vs Concurrent Users: 0.6 %

CPU Utilization: ?(unknown, no issued)29,512 tpmC / 5.83 tps= 5,062 tpmC/tps

56

3.1.8 K-Card

M80 (4-way, 4GB, 34,588 tpmC) x 9WSBCC , Servlet/JSP, CTG

Visitors a day: 5,323 usersVisitors an hour: 4,500 usersPeak Concurrent Users: 2,800 usersHits a day : 6,198,133 hitsPeak Arrival Rate : 217 tpsRequest Interval : 23 secVisitors vs Concurrent Users: 53 %

CPU Utilization: (see left bottom side grahp)311,292tpmC / 217 tps= 1,435 tpmC/tps

[2nd machine]Peak Arrival Rate : 31.7 tpsPeak Concurrent Users: 600 users34,588 tpmC / 31.7 tps= 1,091 tpmC/tps

[4th machine]Peak Arrival Rate : 50 tps34,588 tpmC / 50 tps= 692 tpmC/tps

CPU Utilization

57

3.1.9 Result of my statistics

tpmC / 1 tps(based on 70% cpu)

58

Analysis Methodology

3.2 Bad Process for Perf. Mgmt.

Proposal Pilot BMTAnalysis/

DevelopingFunction

estTesting with

Small real users

SystemOpen

Monitoring

Unit App.Perf.Test

Perf.Prediction

Perf. DataLogging

Additional App. Develop

ApplyingApp.

WorkloadAnalsys

Experienceddata

CapacityEstimation

BMT methodology

AppTuning

Perf Analysis

Unit App.Perf.Test

Perf.PredictionPerf Analysis

2006.02.01 wonyoung lee for web based system performance analysis mathematical approach on

Documents

response time graphcomparison

requestresponse system

time thinktimethroughputtps

time thinktimeactiveuser

requestresponse system

meaning of active user

concurrent user equation2

active users