performance-tuning the spring petclinic sample application

© 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.

Performance-tuning the «Spring Pet Clinic» sample application

By Julien Dubois

Why have Java developers chosen Spring?

DI

AOP

TX

CoreModel

J(2)EE usability

Testable, lightweightmodel for

programming

Application Portability

Powerful Service Abstractions

Deployment Flexibility

Spring

Deploy to Cloud or on premise

Big, Fast,

FlexibleData Web,

Integration,Batch

CoreModel

GemFire

Spring Stack

DI AOP TX JMS JDBC

MVC Testing

ORM OXM Scheduling

JMXREST Caching Profiles Expression

Spring Framework

HATEOAS

JPA 2.0 JSF 2.0 JSR-250 JSR-330 JSR-303 JTA JDBC 4.1

Java EE 1.4+/SE5+

JMX 1.0+WebSphere 6.1+

WebLogic 9+

GlassFish 2.1+

Tomcat 5+

OpenShift

Google App Eng.

Heroku

AWS Beanstalk

Cloud FoundrySpring Web Flow Spring Security

Spring Batch Spring Integration

Spring Security OAuth

Spring Social

Twitter LinkedIn Facebook

Spring Web Services

Spring AMQP

Spring Data

Redis HBase

MongoDB JDBC

JPA QueryDSL

Neo4j

GemFire

Solr Splunk

HDFS MapReduce Hive

Pig Cascading

Spring for Apache Hadoop

SI/Batch

Spring XD

Learn More. Stay Connected.

Talk to us on Twitter: @springcentralFind session replays on YouTube: spring.io/video

About me• Julien Dubois• From Paris, France• Co-authored «Spring par la pratique» (French

book on Spring)• Former SpringSource France director• Director of Consulting at Ippon Technologies• Follow me @juliendubois

6

Goals of this presentation• Make a specific application run faster• Learn classic issues with Spring Web

applications• Learn our methodology to solve

performance issues

7

More information on this presentation• This presentation started with a series of blog posts:

http://blog.ippon.fr/tag/spring-petclinic/• You can run your own tests, every step is cleanly commited

on Github:https://github.com/jdubois/spring-petclinic

8

What’s new since the blog entries?• Java 7• I got a brand new MacBook :-)

–Does a SSD really matters?–Does the new core i7 matters?

9

10

Introducing the Spring Pet Clinic sample

application

Do you remember the Sun Java Pet Store?• Used to explain how you

could build an incredibly complex application

• On a very expensive application server

• With an horrible-looking result

• When your customer only wanted a simple Web app

11

The Spring Pet Clinic• Is here to demonstrate Spring

features and best practices• Is a very classical MVC-style

application (no REST + funky JS library)

• Is still a good excuse to use cute dogs in your presentation

12

More information on the Spring Pet Clinic

• Code:https://github.com/SpringSource/spring-petclinic/

• Short presentation:https://speakerdeck.com/michaelisvy/spring-petclinic-sample-application

13

14

#1Putting the application

in production

Moving to production

• Be as close as possible to a «production environment»–Remove «debugging» logs [ Code ]–Use a «real» database [ Code ]

• For this presentation our environment isn’t perfect–Everything runs on the same machine–Even the presentation itself :-)

15

Memory tuning• We’re running the application from the Maven Tomcat plugin

–Of course Maven takes some resources• Still the application runs with a small memory footprint

–export MAVEN_OPTS="-XX:MaxPermSize=64m -Xms128m -Xmx128m"

• Why not use more memory?–It’s easier to spot memory problems–We don’t need it (we’re not using a big fat Java EE server!)

16

17

#2Creating a JMeter test

Simulate load• JMeter will simulate users

–500 users–In a «realistic» way

• Use a timer• Check «clear cookies each iteration?»

18

Results

• 250 req/sec• Lots of errors• Quickly ends with a crash...

19

20

#3Profiling

Does profiling distort results?• Short answer : not a lot

–We’ll see this when the application runs better• We have used JProfiler and VisualVM in production many

times–Often caused less trouble than official «production monitoring

tools» like HP Diagnostics–Only way to find problems and understand what’s going on–Easy to install

21

Choose your weapon• VisualVM

–Free & lightweight–Included in the JDK

• JProfiler–Best tool we’ve used

• YourKit–Easy to use & cheap

22

First culprit : Dandelion• More specifically, Dandelion module for

DataTables–https://github.com/dandelion/dandelion-datatables

• Nice & fun JSP taglib for rendering JQuery DataTables–Has a memory leak in this version–Is now corrected (extremely quick response time

from the Dandelion team)• So we switched to «plain old JQuery

DataTables» [ Code ]

23

Second culprit : the HTTP Session• Using HTTP Sessions can be helpful

–Developer productivity–Data security

• Going stateless is often better–Uses less RAM–More scalable

24

Going stateless• Example : the PetController.processUpdateForm() method

–Use Spring MVC REST to get more information from the URL–Fetch again the missing data

• We said more scalability, not more performance (for the moment)

• With some work, the whole application can become stateless [ Code ]

25

Results

• 500 req/sec• A few errors• That’s a lot better than before!

26

27

#4Tomcat tuning

Tomcat tuning• This is the easiest step

–Migrate to the latest version of the Tomcat plugin–Use the new NIO connector

• org.apache.coyote.http11.Http11NioProtocol• [ Code ]

28

29

• 950 req/sec• No errors• Victory :-)

Results

30

#5Removing

locks

Locks• Locks are blocked threads

–Aka «there’s a synchronized somewhere»• They obviously cause scalability problems• Let’s launch our profiler again!

31

First culprit : commons DBCP• Commons DBCP does a lot of synchronization• We replaced it with Tomcat JDBC

–New JDBC pool for Tomcat, for highly concurrent systems• But we have kept the pool size at 8

–So obviously we will still have locks–Making it bigger hinders performance

• If the database was on a separate server, we would use a bigger number• The database is always the problem• [ Code ]

32

Second culprit : Webjars• WebJars are client-side libraries (JQuery, etc...) packaged as Jar

files–http://www.webjars.org/–Those libraries are loaded from the classloader

• Which does a big lock each time it is accessed• So if you have a lot of libraries this can be a real problem

• There are many solutions to this issue–Cache those resources with a proxy server–Use another solution for loading those libraries (Bower ?)–[ Code ]

33

Third culprit : the monitoring Aspect• This aspect was introduced on purpose• We should simply disable it [ Code ]

34

Results• 1020 req/sec, still without any error• Only 10% performance increase

35

36

#6JDBC

vs JPA

3 profiles are available• Do you use Spring profiles?• Choose between

–JDBC–JPA–Spring Data

37

Switch to JPA• Change the profile in the web.xml file• The application is a little bit slower

–904 req/sec–After an initial peak, we see the application hanging and then

starting again -> looks like a GC issue

38

First problem : Memory• JPA uses more PermGen, and more memory in general than

JDBC• Let’s boost the application memory

–export MAVEN_OPTS="-XX:MaxPermSize=128m -Xms256m -Xmx256m"

–1103 req/sec ! Better than JDBC !• That’s a different result from our original blog entry

–Looks like we need more memory with JDK 7...

39

Second problem : SQL requests• Run MySQL Workbench

–With JDBC we had more than 30 000 SQL requests per second–With JPA we have 15 000 SQL requests per second

• Why isn’t performance better when we have only half as many requests?

40

41

#7Lazy

loading

Lazy loading & Open Session In View• Pet.visits is an EAGER collection

–This is often a bad idea to eagerly fetch collections• Let’s configure it with the default, lazy-loaded mode• To prevent the dreaded LazyInitializationException we’ll use

the Open Session In View pattern–Not the «perfect» solution, which would require more refactoring–Good enough in this situation, and easy to implement–[ Code ]

42

Results• 1165 req/sec, still without any error• 10 000 SQL req/sec• We have beaten the JDBC version!

43

44

#8Adding cache

Hibernate Caching• We have used Ehcache• We have put Hibernate’s @Cache annotation everywhere

–Owners, Pets, Visits, etc...

45

Spring caching• For more complex queries, we used Spring’s cache

abstraction–Adds a caching aspect around slow methods

• Allowed us to remove the very slow «LIKE» in JpaOwnerRepositoryImpl.findByLastName()

• We could also have used Hibernate’s query cache for the same result

• [ Code ]

46

Results• 1240 req/sec, still without any

error• But can we do better?

47

48

#9Extreme testing

Extreme testing• If we look at the JMeter report, we have reached JMeter’s

limits–Adding more users (=threads) isn’t doing a big difference

• The solution is to do more with the existing threads–Removed the counter–Decreased the ramp-up period to 5 seconds–Run without any listeners (or in command-line)

49

Results• We achieved 3000 req/sec• Memory stayed stable

–We even let it run until we had 100 000 user sessions• We had to increase the JDBC pool size to 30

–Still a small number of connexions–Pushing it to 40 decreases performance

50

51

Final notes

Does JDK 7 helps?• Tested the new G1 Garbage Collector

–export MAVEN_OPTS="-XX:MaxPermSize=128m -Xms256m -Xmx256m -XX:+UseG1GC"

–3% to 5% slower–Needs a bigger heap to work correctly–Doesn’t do any miracle

52

Does the new MacBook Pro helps?• Compared to my original blog posts, with my old MacBook

–We still have the memory issues (until solved)–We still have the HTTP errors (until solved)–Results are 10% to 20% better until the last,

«extreme testing» phase• If you write inefficient code, a faster

CPU and a SSD won’t save you

53

54

#1 Put the application in

production

#2 Create a JMeter test

#3 Profile

#4 Tune the application

server

#5 Remove locks

#7 Lazy loading

#6 JDBC vs JPA

#8 Add cache

55

Questions &

Answers

@julienduboisLate question?