performance tuning grails applications springone 2gx 2014

© 2014 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Performance tuningGrails applications

by Lari Hotari @lhotari

"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts

of their programs, and these attempts at efficiency actually have a strong negative impact when debugging

and maintenance are considered. We should forget about small efficiencies, say about 97% of the time:

premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."

- Donald Knuth, 1974

Mature performance optimisation

• Find out the quality requirements of your solution. Keep

learning about them and keep them up-to-date. It's a

moving target.

• Keep up the clarity and the consistency of your solution.

• Don't introduce accidental complexity.

• Don't do things just "because this is faster" or

someone thinks so.

• Start doing mature performance tuning and optimisation

today!


How do we define application performance?

Performance aspects

• Latency of operations

• Throughput of operations

• Quality of operations - efficiency, usability,

responsiveness, correctness, consistency, integrity,

reliability, availability, resilience, robustness,

recoverability, security, safety, maintainability

Amdahl's law

Little's law

MeanNumberInSystem = MeanThroughput * MeanResponseTime→

MeanThroughput = MeanNumberInSystem / MeanResponseTime

L = λW


Lari's Grails Performance Tuning Method™

TL;DR

Lari's Grails Performance Tuning Method™• Look for 3 things:

• Slow database operations - use a profiler that shows SQL

statements

• Thread blocking - shows up as high object monitor usage

in the profiler

• Exceptions used in normal program flow - easy to check

in profiler

• Pick the low hanging fruits

• Find the most limiting bottleneck and eliminate it

• Iterate


The goals of Grails application tuning

What are we aiming for?

What's the goal of performance tuning?

• The primary goal of performance tuning is to assist in

fulfilling the quality requirements and constraints of your

system.

• Meeting the quality requirements makes you and your

stakeholders happy: your customers, your business

owners, and you the dev&ops.

Performance - Quality of operations

• efficiency

• usability

• responsiveness

• correctness

• consistency / integrity /

reliability

• availability

• resilience / robustness /

recoverability

• security / safety

• maintainability

Operational efficiency

• Tuning your system to meet it's quality requirements

with optimal cost

• Optimising costs to run your system - operational

efficiency


The continuous improvement strategy for performance tuning anything

How do you succeed in performance tuning?

Performance tuning improvement cycle

• Measure & profile o start with the tools you have

available. You can add more tools

and methods in the next iteration.

• Think & learn, analyse and plan

the next changeo find tools and methods to

measure something in the next

iteration you want to know about

more

• Implement a change

Iterate, Iterate, Iterate

• Iterate: do a lot of iterations and change one thing at a

time

• learn gradually about your system's performance and

operational aspects

Feedback from production

• Set up a different feedback cycle for production

environments.

• Don't forget that usually it's irrelevant if the system

performs well on your laptop.

• If you are not involved in operations, use innovative

means to set up a feedback cycle.


More specific derived goals

If your requirement is to lower latency

• Amdahl's law - you won't be able to effectively speed up a

single computation task if you cannot parallellize it.

• In an ordinary synchronous blocking Servlet API

programming model, you have to make sure that the use

of shared locks and resources is minimised.

• Reducing thread blocking (object monitor usage) is a key

principle for improving performance - Amdahl's law

explains why.

• The ideal is lock free request handling when synchronous

Servlet API is used.

Understand Little's law in your context

• With Little's law you can do calculations and reasoning

about programming models that fit your requirements

and available resources

• the traditional Servlet API thread-per-request model

could fit your requirements and you can still make it

"fast" (low latency) in most cases.

Cons of the thread-per-request model in the light of Little's law and Amdahl's law

• From Little's law: MeanNumberInSystem =

MeanThroughput * MeanResponseTime

• In the thread-per-request model, the upper bound for

MeanNumberInSystem is the maximum for the number of

request handling threads. This might limit the throughput of

the system, especially when the response time get higher

or request handling threads get blocked and hang.

• Shared locks and resources might set the upper bound to

a very low value. Such problems get worse under error

conditions.

Advantages of thread-per-request model

• We are used to debugging the thread-per-request model

- adding breakpoints, attaching the debugger and going

through the stack

• The synchronous blocking procedural programming

model is something that programmers are used to

doing.

• There is friction in switching to different programming

models and paradigms.

KillerApp for non-blocking async model

• Responsive streaming of a high number of clients on a

single box

• continuously connected real-time apps where low-

latency and high availablity is a requirement

• limited resources (must be efficient/optimal)


Profiling concepts and tools

JVM code profiler concepts

• Sampling

• statistical ways to get information about the execution using JVM

profiling interfaces with a given time interval, for example 100

milliseconds. Statistical methods are used to calculate values based

on the samples.o Unreliable results, but certainly useful in some cases since the

overhead of sampling is minimal compared to instrumentationo Usually helps to get better understanding of the problem if you learn

to look past the numeric values returned from measurements.

• Instrumentationo exact measurements of method execution details

Load testing tools and services

• Simple command line tools

• wrk https://github.com/wg/wrk

• modern HTTP benchmarking toolo has lua scripting support for doing things like

verifying the reply

• Load testing toolkits and service providers

• Support testing of full use cases and stateful flows

• toolkits: JMeter (http://jmeter.apache.org/),

Gatling (http://gatling.io/)

https://github.com/wg/wrk

http://jmeter.apache.org/

http://gatling.io/

Common pitfalls in profiling Grails

• Measuring wall clock time

• Measuring CPU time

• Instrumentation usually provides false results because

of JIT compilation and other reasons like spin locks

• lack of proper JVM warmup

• Relying on gut feeling and being lazy

Ground your feet

• Find a way to review production performance graphs regularly,

especially after making changes to the system

• system utilisation over time (CPU load, IO load & wait, Memory

usage), system input workload (requests) over time, etc.

• In the Cloud, use tools like New Relic to get a view in operations

• CloudFoundry based Pivotal Web Services and IBM Bluemix

have New Relic available

• In the development environment, use a profiler and debugger to

get understanding. You can use grails-melody plugin to get

insight on SQL that's executed.

Grails - The low hanging fruit

• Improper JVM config

• Slow SQL

• Blocking caused by caching

• Bad regexps

• Unnecessary database transactions

• Watch out for blocking in the Java API: Hashtable

Environment related problems

• Improper JVM configuration for Grails apps

• out-of-the-box Tomcat parameters

• a single JVM running with a huge heap on a big box o If you have a big powerful box, it's better to run

multiple small JVMs and put a load balancer in front

of them

Example of proper Tomcat config for *nix Create a file setenv.sh in tomcat_home/bin directory:

1 export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_60 2 export LC_ALL=en_US.UTF-8 3 export LANG=en_US.UTF-8 4 CATALINA_OPTS="$CATALINA_OPTS -server -noverify" 5 CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=256M -Xms768M -Xmx768M" # tune heap size 6 CATALINA_OPTS="$CATALINA_OPTS -Djava.net.preferIPv4Stack=true" # disable IPv6 if not used 7 # set default file encoding and locale 8 CATALINA_OPTS="$CATALINA_OPTS -Dfile.encoding=UTF-8 -Duser.language=en -Duser.country=US" 9 CATALINA_OPTS="$CATALINA_OPTS -Duser.timezone=CST" # set default timezone 10 CATALINA_OPTS="$CATALINA_OPTS -Dgrails.env=production" # set grails environment 11 # set timeouts for JVM URL handler 12 CATALINA_OPTS="$CATALINA_OPTS -Dsun.net.client.defaultConnectTimeout=10000 -Dsun.net.client.defaultReadTimeout=10000" 13 CATALINA_OPTS="$CATALINA_OPTS -Duser.dir=$CATALINA_HOME" # set user.dir 14 export CATALINA_OPTS 15 export CATALINA_PID="$CATALINA_HOME/logs/tomcat.pid"

JVM heap size

• Assumption: optimising throughput and latency on the cost of

memory consumption

• set minimum and maximum heap size to the same value to

prevent compaction (that causes full GC)

• look at the presentation recording of the "Tuning Large scale

Java platforms" by Emad Benjamin and Jamie O'Meara for more.

• rule in the thumb recommendation for heap size: survivor space

size x 3...4 and don't exceed NUMA node's local memory size

for your server configuration (use: "numactl --hardware" to find

out Numa node size on Linux).

The most common problem: SQL

• SQL and database related bottlenecks: learn how to profile

SQL queries and tune your database queries and your

database

• grails-melody plugin can be used to spot costly SQL

queries in development and testing environments.

Nothing prevents use in production however there is a

risk that running it in production environment has

negative side effects.

• New Relic in CloudFoundry (works for production

environments)

Use a non-blocking cache implemention

• Guava LoadingCache is a good candidate

https://code.google.com/p/guava-libraries/wiki/CachesExplained

• "While the new value is loading the previous value (if any)

will continue to be returned by get(key) unless it is evicted.

If the new value is loaded successfully it will replace the

previous value in the cache; if an exception is thrown while

refreshing the previous value will remain, and the exception

will be logged and swallowed." (

http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/cache/LoadingCache.html#refresh(K))






Some regexps are CPU hogs

https://twitter.com/lhotari/status/474591343923449856

https://twitter.com/lhotari/status/474591343923449856

Verify regexps against catastrophic backtracking

• Verify regexps that are used a lot

• use profiler's CPU time measurement to spot

• search for the code for candidate regexps

• Use a regexp analyser to check regexps with different input size

(jRegExAnalyser/RegexBuddy).

• Make sure valid input doesn't trigger "catastrophic backtracking".

• Understand what it is.

• http://www.regular-expressions.info/catastrophic.html

• "The solution is simple. When nesting repetition operators, make

absolutely sure that there is only one way to match the same

match"

http://www.regular-expressions.info/catastrophic.html

Eliminate unnecessary database transactions in Grails

• should use "static transactional = false" in services that

don't need transactions

• Don't call transactional services from GSP taglibs (or

GSP views), that might cause a large number of short

transactions during view rendering

JDK has a lot of unnecessary blocking

• java.util.Hashtable/Properties is blocking

• these block:

System.getProperty("some.config.value","some.defaul

t"), Boolean.getBoolean("some.feature.flag")

• Instantiation of PrintWriter, Locale, NumberFormats,

CurrencyFormats etc. , a lot of them has blocking

problems because System.getProperty calls.

• Consider monkey patching the JDK's Hashtable class:

https://github.com/stephenc/high-scale-lib

https://github.com/stephenc/high-scale-lib

Misc Grails tips

• Use singleton scope in controllers

• grails.controllers.defaultScope = 'singleton'

• default for new apps for a long time, might be

problem for upgraded apps

• when changing, make sure that you previously didn't

use controller fields for request state handling (that

was ok for prototype scope)

• Use controller methods (replace closures with

methods in upgraded apps)


Tools for performance environments

Simple inspection in production environments

• kill -3 <PID> or jstack <PID>

• Makes a thread dump of all threads and outputs it to

System.out which ends up in catalina.out in default

Tomcat config.

• the java process keeps running and it doesn't get

terminated

Java Mission Control & Flight Recorder

• Oracle JDK 7 and 8 includes Java Mission Control since

1.7.0_40 .

• JAVA_HOME/bin/jmc executable for launching the client

UI for jmc

• JMC includes Java Flight Recorder which has been

designed to be used in production.

• JFR can record data without the UI and store events in

a circular buffer for investigation of production

problems.

JFR isn't free

• JFR is a commercial non-free feature, available only in

Oracle JVMs (originally from JRockit).

• You must buy a license from Oracle for each JVM using

it.

• "... require Oracle Java SE Advanced or Oracle Java

SE Suite licenses for the computer running the

observed JVM" ,

http://www.oracle.com/technetwork/java/javase/documentation/java-se-product-editions-397069.pdf

, page 5



Controlling JFR

• enabling JFR with default continuous "black box"

recording:

• Runtime controlling using jcmd commands

• help for commands with jcmd <pid> help JFR.startjcmd <pid> help JFR.stopjcmd <pid> help JFR.dumpjcmd <pid> help JFR.check

export _JAVA_OPTIONS="-XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=true"


Demo


wrk http load testing tool sample output

1 Running 10s test @ http://localhost:8080/empty-test-app/empty/index 2 10 threads and 10 connections 3 Thread Stats Avg Stdev Max +/- Stdev 4 Latency 1.46ms 4.24ms 17.41ms 93.28% 5 Req/Sec 2.93k 0.90k 5.11k 85.67% 6 Latency Distribution 7 50% 320.00us 8 75% 352.00us 9 90% 406.00us 10 99% 17.34ms 11 249573 requests in 10.00s, 41.22MB read 12 Socket errors: connect 1, read 0, write 0, timeout 5 13 Requests/sec: 24949.26 14 Transfer/sec: 4.12MB

check latency, the max and it's distribution

Total throughput

https://github.com/lhotari/grails-perf-testapps/empty-test-app

https://github.com/lhotari/grails-perf-testapps/empty-test-app


Questions?


Thanks!Lari Hotari @lhotari

Pivotal Software, Inc.