Look Inside Your Portal

Portal Specialist / Consultant

IBA CZ, s. r. o.

Monitoring and Troubleshooting Techniques for Java and Liferay Portal

Aleš Rybák

organization

● workshop

● 90 minutes

– quick intro (with images copying meanwhile)

– hands on

● problems

– solve them with assistant ASAP

– if problems are still in place join somebody and work together

prerequisites

● your own HW

● 4GB+ RAM is optimal

● virtual images

– Oracle VirtualBox 4.2.18

– CentOS 6.4 32-bit

– Oracle Java 1.6

– PostgreSQL database

– Liferay Portal 6.1.2 CE

● credentials

operating systemuser: testpass: test

liferay portaluser: test@liferay.com pass: test

questions

targets

● look onto / into Liferay Portal using different techniques and from different angles

● learn how to monitor it

● learn how to solve some specific problems

● learn how to optimize some performance aspects

Liferay Portal is standard J2EE application. It is running on application server which is running in JVM which is running as standard application in

operating system.

the “onion” model

the “onion” model – OS

● process monitoring

● system load

● system resources

– CPU

– mem

– HDD

– network

– ...

the “onion” model – JVM

● JMX

● GC logs

● memory dumps

● stack

● profiling

● debugging

● ...

the “onion” model – application server

● logs

● interceptors

● admin console

● monitoring apps

● specific tools

the “onion” model – liferay portal

● administration console

– memory

– admin tasks

– properties

– scripting console

● mbeans

– ehcache

– hibernate

– liferay

– …

hands on

liferay startup

● cd liferay/bundles/tomcat

● bin/startup.sh

● tail -f logs/catalina.out

– best in new terminal

● available at http://localhost:8080

● bin/shutdown.sh

operating system

● desktop tools

– good for local usage, not for server monitoring

– conky

– gkrell

– …

● command-line tools

– free, df, du

– ps

– top (htop)

– lsof

– iostat

– vmstat

– netstat

– ...

operating system – interesting commands

● ps aux | grep tomcat

● top (keys h, F/O)

● htop

● lsof -p $PID

● iostat (optionally with -m)

● vmstat -t 1

● netstat -nlpt

● netstat -nlpt | grep $PID

operating system - vmstat

● Procs – r: Total number of processes waiting to run

● Procs – b: Total number of busy processes

● Memory – swpd: Used virtual memory

● Memory – free: Free virtual memory

● Memory – buff: Memory used as buffers

● Memory – cache: Memory used as cache.

● Swap – si: Memory swapped from disk (for every second)

● Swap – so: Memory swapped to disk (for every second)

● IO – bi: Blocks in. i.e blocks received from device (for every second)

● IO – bo: Blocks out. i.e blocks sent to the device (for every second)

● System – in: Interrupts per second

● System – cs: Context switches

● CPU – us, sy, id, wa, st: CPU user time, system time, idle time, wait time

java virtual machine

● what to monitor

– processes

– memory

● garbage collecting

– threads

● profiling

● debugging

● tools

– jps

– jstat

– jinfo

– jstack

– jcmd

– jconsole, jvisualvm

– gcviewer

– jmap, jhat, MAT

jvm tools – iteresting commands

● jps -l

● jps -v

● jstat -gc $PID 2000

● jinfo $PID

● jmap -histo $PID

● jstack $PID

JMX

http://en.wikipedia.org/wiki/File:Jmxarchitecture.png

jconsole

gcviewer

heap dump analysis – acquire dump

jmap -dump:live,format=b,file=heap.bin $PID

heap dump analysis – analyze dump

● Eclipse Memory Analyzer Tool (MAT)

application server

● logs

● specific tools

– apache tomcat

● mbeans

● valves

● interceptors (JDBC, tribes, …)

● conf/server.xml

– …

● monitoring apps

– http://localhost:8080/monitoring

liferay portal (and other applications)

● administration console

– memory

– admin tasks

– properties

– scripting console

– good for basic tasks and for beginners :-)

● mbeans

– ehcache

– hibernate

– liferay

– …

● ad-hoc tools

– spring aspects

– portlet filters

ehcache mbeans

● portal property ehcache.statistics.enabled=true

hibernate mbeans

● portal property hibernate.generate_statistics=true

liferay mbeans

● enable liferay monitoring (see portal properties)

● optionally install monitoring hook (done in the virtual image)

● performance info is visible

– at the end of each page

– via JMX beans

heap dump analysis - hands-on

● http://localhost:8080/tm/

● watch your heap, GC, system memory

● obtain heap dump

– jps -l

– jmap -dump:live,format=b,file=heap.bin $PID

● analyze heap dump

– mat

– File > Open Heap Dump..

– Leak Suspects

links

● iostat, vmstat, netstat...

– www.lmgtfy.com/?q=vmstat+iostat+examples :-)

● java tools – http://docs.oracle.com/javase/7/docs/technotes/tools/

● MAT – www.eclipse.org/mat

● gcviewer - www.tagtraum.com/gcviewer.html

● liferay

– portal props - www.liferay.com/community/wiki/-/wiki/Main/Portal+Properties

– dev guide - www.liferay.com/documentation/liferay-portal/6.1/development

visit uswww.ibacz.eu

follow us@IBACZ

the end