Slide 1

The science of performanceMike CroftHands-on Performance Workshop

Who are you?

Firstly who are you?

I want to know who my audience is so I can avoid teaching Grandma to suck eggs!

Who works with Java EE middleware? Standard Java EE or Spring?Who is a developer?Who is operations or support?2

Who Am I?C2B2 Expert Support consultant

Ex-IBM

Snowboarder

@croft

C2B2 Expert Support consultantThis doesnt mean much I handle all sorts of problems with Java middleware.Includes WebLogic/SOA Suite; Jboss/WildFly; GlassFish/Payara; Tomcatetc

Ex-IBMUsed to work with WebSphere tooStill follow WebSpheres activityLiberty Profile is pretty coolVery intuitive to build a profile

SnowboarderI love speed!Dodgy collarbone to prove itWondering if Id rather be snowboarding?The answer is always yes

@croftTweet me!

3

Why are we here?Hands-on performance!Hands on what?See what tools are availableGet experience of using some toolsLearn how to apply knowledge

Why do you want to be here?

AgendaIntroductionEnvironment setupPerformance OverviewCollecting DataPresentationPracticalHow do we interpret the data?ToolsExample app analysis

What Wont We Talk About?ProfilingCovered by Simon Maple of ZeroTurnaround earlier this weekCode examplesMicrobenchmarksAleksey Shipilev and JEP 230 have that covered :-)http://openjdk.java.net/jeps/230

Extension ActivitiesTwo extra activities if you finish others early

Deliberately left with very little instruction

Use techniques which are most useful to you

Hands-on Performance WorkshopEnvironment Setup

Environment SetupDownload Instructions and follow the first part

Make sure all necessary resources are downloadedMake sure the app is deployed and JMeter can run load through ithttps://s3-eu-west-1.amazonaws.com/devoxx2015/instructions.docxhttps://s3-eu-west-1.amazonaws.com/devoxx2015/extension-activities.docx

Hands-on Performance WorkshopPerformance Overview

Performance Overview

Performance in middleware comes down to:How fast does a single transaction take to execute?

Faster Performance = Happier CustomersFaster Performance = More TransactionsHOWEVERPerformance is the P in R-A-S-P and the other letters should be ignored to your peril!There is always a trade-off to consider!

11

Performance Overview

Performance Factors

Raw Algorithmic PerformanceResource LimitationsNot enough cpu, disk, memoryResource ContentionLocks IO LatencyNetwork, Disk

Performance vs Latency

Performance Factors = what things do you have that can make you run faster?Latency Factors = what things do you have which might slow you down?

How to maximise performance and minimise latency

Raw Algorithmic PerformanceJust improving the way your algorithms workVenkat has done a few talks about lazy evaluation in Java 8Rather than process an entire collection for each operation, it only does as much work is needed to satisfy the final operationUse someone elses algorithmJSR 107 JCache is coming. The simplest implementation of JCache is just a ConcurrentHashMap, but writing the code to manage the hashmap yourself is asking for trouble!Hazelcast, Infinispan, Coherence, etc etc all implement the JCache standard, so they will manage your data, ensure best performance for synchronisation and help you avoid excessive memory usage. Because its a standard API, you can be loosely coupled enough to swap out your provider by changing your maven dependencyResource LimitationsNot enough cpu, disk, memory

Think of lunchtime!We have a few stations to serve people one of two options, but there is a limit to the amount of food that can go in each container. Once the container is finished, we have to fetch more.We cant fetch more too soon, because it will get cold and flies might get on it, so we have a stop-the-food pause Remedy here is to add another station which can keep the food warm and serve it at the same time.We dont want too many food stations, because they are expensive (wages, energy, etc) and if there are too many then there wont be enough people to eat the food and keep all the stations busy.

Resource ContentionLocks Back to the food station!If there are two lines with two types of food to share then we can serve both lines in parallelIf there is only one serving spoon, then we can only serve the two lines concurrentlyWhile the Chicken line is being served, they have a lock on the serving spoonIf this lock is held for too long, it could slow down performance a lotHow is the spoon cleaned? A bucket with hot soapy water and a cloth at the station is the quickest wayIf the spoon has to go back to the kitchen to be cleaned, the lock will be held for far too long!IO LatencyNetwork, DiskHad a customer who was writing logs to SAN based storage, with a massive Gigabit pipe. They were running on SOA Suite, which has a lot of moving parts that no-one understands, so it was very difficult to convince the network guys that they needed to take a look at their SAN configuration. They had a very low latency Gigabit pipe, so they thought it couldnt possibly be their fault.It turned out that the network was configured to send data from the SOA Suite domain to the SAN by going through a very low capacity switch, which was introducing latency of seconds for each write!

Also had a customer who wanted to use log-based monitoring, by using logstash. Their solution (in production!) was to write logs directly to and AWS instance, which was located in Ireland

12

Performance Overview

Latency Factors

Network DistanceNetwork ReliabilityData SizeOperation GranularityResource ContentionJVM GC

LATENCY = Time delay in requesting an operation and it being initiatedKey factor in large scale distributed applicationsTypically not taken into account during development

13

Performance Overview

Lots of controls, with some knowledge of how each control worksUnlikely to have full understanding of how each control works individuallyHighly unlikely to have full understanding of how all work togetherImpossible (?) to have full understanding of how all work together with your code on top!14

Performance Overview

Hands-on Performance WorkshopCollecting Data

Collecting Data

Garbage CollectionVerbose GCHeap sizeNew size and old sizeBefore collectionAfter collectionPause time

Flight Recorder (commercial feature only)17

Collecting Data

Thread dumpsKill -3JStackThread stateWaitSleepBlockedRunningFull stack trace

Simon Maple earlier this week:Thread dumps make very good bedtime reading, cause theyll put you to sleep!They dont have to!You dont need to be a performance engineering ninja to find some usefulness in a thread dump(Although the right tools help)18

Collecting Data

Heap dumpsEntire contents of the heapVery Large!Can take time to collect on largeheapsCan auto-dump on OOME

Hands-on Performance WorkshopInterpreting Data

Interpreting Data

ToolsLive monitoringVisualVMMissionControlJMX

Historical analysisMemory Analyzer ToolFlight Recorder (commercial)ThreadlogicGCViewer

Tips:Look for whats obvious!Dont spend hours/days/weeks/months looking in the same place, when there are other unexplored areasYou can always (and should) revisit a data set21

Hands-on Performance WorkshopTuning for Performance

StrategyPlanningWhat is your definition of success?What settings are available to tune?Can you prioritise which to tune first?ExecutionTest

PlanningWhat is your definition of success?What settings are available to tune?Can you prioritise which to tune first?ExecutionTest

23