colorado software summit: october 21 – 26, 2007 … · management, monitoring, and deployment of...

Colorado Software Summit: October 21 – 26, 2007

Simon Roberts – Management, Monitoring, and Deployment of Tomcat and JBoss Slide 1

© Copyright 2007, Dancing Cloud Services

Management, Monitoring, and Deployment of Tomcat and JBoss

Simon [email protected]




Agenda

Installation and startup

Management

Web console elements

System and use of MBeans for monitoring/control

J2EE Domain for application monitoring

JMX Command-line access

Monitoring

Graphs, snapshots, and monitors

Connecting JConsole




Agenda

Deployment

Removal/reconfiguration of some services:

Web-console/JMX

HTTP Invokers

Class downloader

Basic clustering of JBoss

Basic load balancing with Apache




Install & Startup




Download/Install

Jboss.com -> Downloads -> Community downloads -> JBoss

application server -> 4.2.1.GA Download -> (sourceforge) jboss-4.2.1.GA.zip

Unzip

(Older versions require JAVA_HOME)




Execute

Execute jboss-4.2.1.GA/bin/run.sh (run.bat)

bind to appropriate IP address

Browse to MyHost:8080/

Verify inbound firewall access to

HTTP/HTTPS (or 8080 by default)

Might require other firewall holes for other

services




Default Homepage




Management




Web Console

Java applet at:myserver:port/web-console

Install java plugin in your browser

Typically link or copy:

$JAVA_HOME/jre/plugin/i386/ns7/libjavaplugin_oji.so

to your plugins directory, perhaps:

/usr/lib/browser-plugins

.../firefox/plugins

(or whatever)




Web Console




Web Console Views

System

[Monitoring]

J2EE Domains

[AOP]




System View

JMX Beans

Provide information or control

Attributes may be read/write

Operations may be invoked

Grouped by “Domain Name”

e.g. jboss.system

Bean has a “type” name

e.g. type=Server and type=ServerInfo

Subset of this information and control is provided by the JMX console




ServerInfo JMX Bean




Server JMX Bean




Useful MBeans and Elements

jboss.system:type=Server

JVM, Build, Version info

shutdown, runGarbageCollector operations

jboss.system:type=ServerInfo

Thread count, IP address, processors, memory/free memory, JVM, OS version

listThreadCpuUtilization, listThreadDump,

listMemoryPools

jboss:service=JNDIView

list




Useful MBeans and Elements

jboss.jca:service=ManagedConnectorPool,

name=DataSourceName

InUseConnectionCount

MinSize

ConnectionCreatedCount

MaxSize




J2EE Domains

Observe individual applications

Deployment descriptor

Number of calls to servlet/jsp

Calls to EJB methods

Max/min/count of EJBs method-ready

Minimum/maximum response time




Command Line Access

bin/twiddle.sh \--server=MyServer:1099 get \jboss.system:type=ServerInfo

Assumes port 1099 is open

Can also invoke operations:

bin/twiddle.sh -s Christine invoke \ jboss.system:type=Server shutdown

Clearly, none of this is safe for remote access

Block port 1099 by firewall

But HTTP Invokers bypass this




Connecting Jconsole

JConsole might provide advantages over web console

e.g. more detailed memory investigation

Enable this in the bin/run.conf file or edit run.bat

Add:

-Dcom.sun.management.jmxremote

Note: in run.conf, be sure to get this inside the quote marks




Monitoring




Graphing an MBean Property

Find the MBean's domain in:

Web Console

System

JMX Mbeans

Expand the domain, then expand the bean

Right click for context menu

Select graph




MBean Graph

Select jboss.system:type=ServerInfo and graph TotalMemory




Graphables

Graphs can be drawn for numeric bean attributes

e.g. memory use/free, connection count

Even silly ones like port numbers

You can add your own MBeans and graph them too

Be careful interpreting Free Memory




Snapshots

Graphs are great, but they go away

Snapshots collect data for later use

Right click on a numeric data item in an MBean

Select Create Snapshot

Fill in the time interval in milliseconds

Press Create




Snapshot Preparation




Snapshot Use

After creation Start Snapshot

Later, Graph Dataset




Snapshot Recovery

Can also control snapshot and retrieve data using the MBean created for the

snapshot

Click on JMXBeans, search for

jboss.snapshot:*

Select relevant bean to start, stop, and

retrieve data as text




Monitoring

In addition to graphs, you can configure

notifications that will advise if a numeric

attribute changes in particular ways

Right click on the attribute, Create Monitor

Provide a name, threshold value, period, and

enable the monitor

Ensure the period won't miss a spike

Don't forget to enable it

Console Alert sends message to log

You can arrange to send email




Create Monitor




Deployment




Protecting Services

Some services should be password protected

Others should be shut down or reconfigured

Web console, JMX services

HTTP Invoker

Class download service

Database choice




Remove Console Services

JMX console:

Remove (default)/deploy/jmx-console.war

Also disables JMX/MBean functions in Web

console.

For web console, remove: (default)/deploy/management/console-mgr.sar




Password Protection

Alternative to removing, require login

edit (default)/deploy/jmx-console.war/WEB-INF/web.xml

uncomment provided <security-

constraint> element

<login-config> and <security-role> elements are already provided

edit (default)/deploy/jmx-console.war/WEB-INF/jboss-web.xml

uncomment provided <security-domain>

element




Security Constraint

<security-constraint> <web-resource-collection> <web-resource-name>HtmlAdaptor</web-resource-name> <description> An example security config that only allows users with the role JBossAdmin to access the HTML JMX console web application </description> <url-pattern>/*</url-pattern> <http-method>GET</http-method> <http-method>POST</http-method> </web-resource-collection> <auth-constraint> <role-name>JBossAdmin</role-name> </auth-constraint></security-constraint>




Password Change

By default, security is referred, by jboss-web.xml, to a policy named

“jmx-console”

Defined in (default)/conf/login-config.xml

as property file based

Inevitably, default user/password are well known (admin/admin!)




Username Password Files

Create/change user=password entries in the file:

(default)/conf/props/jmx-console-users.properties

Ensure matching user=role entry exists in file:

(default)/conf/props/jmx-console-roles.properties

Role required in security constraint was

JBossAdmin




HTTP Invoker

Used for tunneling when poor firewall administration forces it

Best removed if not needed

Simply remove http-invoker.sar from the deployment directory




Class Download Service

Used for dynamic loading of stubs,

polymorphic argument types, etc.

Powerful RMI feature permits object oriented

distributed computing

Not just object-based

But largely unnecessary if classes are pre-

installed on all participating systems

Allowing attackers to download your classes,

or worse, resources, is bad

Resources include the username/password files!




Removing Class Download Service

Edit conf/jboss-service.xml

Remove the entry:

<mbean

code=”org.jboss.web.WebService”name=”jboss:service=WebService>

...

</mbean>

Editing the contents can provide more

control if you don't want to disable class downloading completely




Basic JBoss Cluster

Essentially provided out of the box

Run the “all” configuration

Multiple machines will cluster

For demo, run on single machine

Create duplicate configuration called “all2” or similar

Arrange to use alternate port numbers (base + 100)

In conf/jboss-service.xml uncomment the element<mbean code=”org.[...].ServiceBindingManager”[...]




Session Replication

Provided by default

Be sure to mark application as

<distributable/> to enable replication




Load Balancing

Use Apache as front end load balancer

Don't forget that LB is single point of

failure — monitor it carefully and treat it well!

Install mod_jk-apache.?.??.?.so into apache2/modules

Configure the jk module

LoadModule jk_module modules/mod_jk-apache-2.0.55.so




Configure jk module

# Locate workers.propertiesJkWorkersFile conf/workers.properties# Location of jk logsJkLogFile logs/mod_jk.log# jk log levelJkLogLevel info# log formatJkLogStampFormat "[%a %b %d %H:%M:%S %Y] "# JkOptions indicate to send SSL KEY SIZE, JkOptions +ForwardKeySize +ForwardURICompat\ -ForwardDirectories# JkRequestLogFormat set the request format JkRequestLogFormat "%w %V %T"JkMount /mycluster/* mybalancerJkMount /mycluster mybalancer




Configure Workers

ajp13worker.list=mybalancerworker.mybalancer.type=lbworker.mybalancer.balance_workers=worker1,worker2worker.mybalancer.sticky_session=0(ajp13)worker.worker1.type=ajp13worker.worker1.host=localhostworker.worker1.port=8009worker.worker1.lbfactor=1(ajp13)worker.worker2.type=ajp13worker.worker2.host=localhostworker.worker2.port=8109worker.worker2.lbfactor=1




Configure Apache

Edit apache2/conf/httpd.conf

In the LoadModule block, insert:

Include conf/jk.conf




Summary

Installation and startup

MBeans via console and JMX command line

J2EE Domain

Graphs, snapshots, and monitors

Connecting JConsole

Removal/reconfiguration of services

Basic clustering and load balancing




Management, Monitoring, and Deployment of Tomcat and JBoss

Simon [email protected]

colorado software summit: october 21 – 26, 2007 … · management, monitoring, and deployment of...

Documents