performance tuning through iterations

54
Performance Tuning a CFML script Gert Franz Rasia GmbH

Upload: gert-franz

Post on 21-Jan-2018

151 views

Category:

Internet


2 download

TRANSCRIPT

Page 1: Performance tuning through iterations

Performance Tuning a CFML script

Gert FranzRasia GmbH

Page 2: Performance tuning through iterations
Page 3: Performance tuning through iterations

WHO AM I?

• Gert Franz

– Involved with Lucee and Railo since day 1

– Into CFML for 10 11 12 13 14 15 16 17 years

– DBA, System architect

– CTO Rasia Ltd, CIO Rasia CH

• Rasia is founding Member of the Lucee association

Page 4: Performance tuning through iterations

MY EVOLUTION

Assembler Basic Clipper Pascal Delphi Java CFML Lucee

Page 5: Performance tuning through iterations

WHAT IS THIS ABOUT

• Having some fun with CFML

• Discrediting long known approaches

• Looking for new ways of solving a given problem

Page 6: Performance tuning through iterations

TASK AT HAND

• The client reported that there was

– An import job that imports an .xlsx file

– Into a database table

– With some special rules

Page 7: Performance tuning through iterations

INITIAL CODE

• The code is around 2 years old

• Takes around 20 minutes to execute

• Simply imports a total of 25000 recordsinto a database with some specific checks

• So nothing really fancy

Page 8: Performance tuning through iterations

STARTING POINT

• Let's have a look at the source code that was used to fulfil the task

Page 9: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

Page 10: Performance tuning through iterations

MAIN POINTS VERSION 1

• Uses a default datasource• Uses lists for lookups• Creates an array to hold fieldnames• Creates a query and imports all into the

query• Uses two QoQ to delete unnecessary

records• Uses a QoQ to add only needed records to

an array• Insert the array into the database table

Page 11: Performance tuning through iterations

IDEAS? PROPOSALS?

Page 12: Performance tuning through iterations

FIRST ITERATION

• Convert everything to CFScript

WHY???

Page 13: Performance tuning through iterations

ITERATION 1

• Use a function to load the XLSX content

• Optimize filling the fields

• Exit early in the cleanUpNumber() function

Page 14: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript, reduced the field building effort

Page 15: Performance tuning through iterations

ITERATION 2

• Replace the main QoQ with a nested struct

Page 16: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

Page 17: Performance tuning through iterations

QUERY OF QUERY EVIL

Page 18: Performance tuning through iterations

QUERY OF QUERY

• Lucee is no database server

• So nothing of the following

– Query analyzer

– Indexes

– Statistics

– Query Optimizations

– etc!

Page 19: Performance tuning through iterations

QUERY OF QUERY

• Avoid if possible at all times!

Page 20: Performance tuning through iterations

ITERATION 3

• Eliminated the evaluate approach

Page 21: Performance tuning through iterations

EVALUATE – SMALLER EVIL

Page 22: Performance tuning through iterations

EVALUATE

• Evaluate sometimes does a lot of parsing

• I mostly use it only on serialized complex objects

• VERY OFTEN SEEN:

<cfset b = evaluate("form.#fieldName#")>

Instead use:

<cfset b = form[fieldname]>

Page 23: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

Page 24: Performance tuning through iterations

ITERATION 4

• Eliminated the SQL statements from getMarketID()

• Avoid too many unnecessary SQL statements

Page 25: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

Page 26: Performance tuning through iterations

ITERATION 5

• Eliminated all QoQ

• Only added stuff when model > 0

Page 27: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

Page 28: Performance tuning through iterations

ITERATION 6

• Why create a query in the first place?

• Why not create the resulting struct right away?

Page 29: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminated creating a query and filling the struct

Page 30: Performance tuning through iterations

ITERATION 7

• What is always the thing that consumes the most time?

• Queries

• Let's get rid of many of them

Page 31: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

Page 32: Performance tuning through iterations

ITERATION 8

• We only fill the struct in cases where it makes sense

• So why filter afterwards, if we can avoid filling these values in, in the first place

Page 33: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Exiting early from filling the struct

Page 34: Performance tuning through iterations

ITERATION 9

• Create a bulk insert, instead of lots of individual inserts

Page 35: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine

9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File

Page 36: Performance tuning through iterations

ITERATION 10

• Instead of nested structs, use composite keys

• Nested structs use lots of recursions

Page 37: Performance tuning through iterations

INTERMEDIATE RESULT

# T (ms) Δ (ms) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine

9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File

10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key

Page 38: Performance tuning through iterations

ITERATION 11

• Replaced lists with arrays

• Generally speaking: Lists are to be avoided if possible

• Used len() instead of isEmpty() and added an exit early strategy

Page 39: Performance tuning through iterations

INTERMEDIATE RESULT

# T (s) Δ (s) Imp. Total Factor Size Remarks

0 658.15 0 0.00% 100.00% 1.00 5'867 B Initial version

1 634.30 23.85 3.62% 96.38% 1.04 5'137 B Changed to CFScript

2 31.363 602.94 95.06% 4.77% 20.98 5'065 B Eliminated QoQ

3 25.26 6.10 19.46% 3.84% 26.06 5'102 B Eliminated evaluate

4 15.154 10.11 40.01% 2.30% 43.43 5'084 B eliminated getMarketID

5 12.37 2.78 18.36% 1.88% 53.20 4'409 B Eliminated all QoQ

6 11.01 1.36 10.99% 1.67% 59.77 4'325 B Eliminating the filling the struct

7 4.11 6.90 62.70% 0.62% 160.21 4'292 B Improved Inserting records

8 3.65 0.46 11.08% 0.56% 180.17 4'680 B Copying into SQL Statements in a single Routine

9 0.65 3.00 82.15% 0.10% 1009.43 4'072 B Bulk import from local File

10 0.41 0.24 36.96% 0.06% 1601.34 3'957 B Replaced a nested struct with another one with a nested key

11 0.32 0.09 21.90% 0.05% 2050.31 4'035 B Replaced isEmpty() with len() and eq 0, eliminated Trim

Page 40: Performance tuning through iterations

FINAL RESULT CHART

0

100

200

300

400

500

600

700

1 2 3 4 5 6 7 8 9 10 11 12

Page 41: Performance tuning through iterations

FINAL RESULT CHART

0

100

200

300

400

500

600

700

1 2 3 4 5 6 7 8 9 10 11 12

Page 42: Performance tuning through iterations

FINAL RESULT CHART

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10 11 12

Page 43: Performance tuning through iterations

FINAL RESULT CHART

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 11 12

Page 44: Performance tuning through iterations

FINAL RESULT CHART

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12

Page 45: Performance tuning through iterations

FINAL RESULT CHART

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10 11 12

Page 46: Performance tuning through iterations

FINAL RESULT CHART

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12

Page 47: Performance tuning through iterations

FINAL RESULT CHART

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1 2 3 4 5 6 7 8 9 10 11 12

Page 48: Performance tuning through iterations

FINAL RESULT CHART

0

0.5

1

1.5

2

2.5

3

3.5

4

1 2 3 4 5 6 7 8 9 10 11 12

Page 49: Performance tuning through iterations

FINAL RESULT CHART

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4 5 6 7 8 9 10 11 12

Page 50: Performance tuning through iterations

FINAL RESULT CHART

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

1 2 3 4 5 6 7 8 9 10 11 12

Page 51: Performance tuning through iterations

AS A PIE CHART

3.6%

91.7%

0.9%1.5% 0.4%0.2%1.0%0.1%0.5%0.0%0.0%

3.6%

0.9%

1.5%

0.4%

0.2%

1.0%

0.1% 0.5%

0.0%0.0%

Page 52: Performance tuning through iterations
Page 53: Performance tuning through iterations

CONCLUSIONS

• DON'T USE QueryOfQuery

• Don't use heavily nested structs

• Avoid lists, use structs or arrays instead

• Avoid too many database calls

Page 54: Performance tuning through iterations

CONCLUSIONS

• If you know the data you're handling, performance tuning is a lot easier

• First generalize in order to understand the code

• Then specialize in order to tune it