CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

MSG Status update

Daniel Rodrigues

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Presentation Summary

• MSG overview• Timeline• ActiveMQ• Msg-publish-simple• Msg-consumer2oracle• Current deployments

– OSG– Gridview

• Next steps

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

MSG overview

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

• Features of an Messaging System:– Flexible architecture:

• Deliver messages, either in point to point (queue)…• … or multicast mode (topics)• Support Synchronous or Asynchronous communication.

– Reliable delivery of messages:• Provide reliability to the senders if required• Configurable persistency / Master-Slave.

– Highly Scalable:• Network of Brokers

• ActiveMQ is an Open Source message broker providing these and many other features.

MSG overview

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

• A real life working example:

MSG overview

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Timeline

• 1st Trimester (Oct/Dez):– Testing of ActiveMQ 4.1, using netcat/bash scripts to test throughput;– Integrated testing with twisted stomp, python framework;

• 2nd Trimester (Jan/Mar):– Simple publishers and python consumers developed by Piotr Nyczyk on place for

OSG;– Msg-publish-simple and msg-consume2oracle (bugfixes + improvements + testing);– Moving to ActiveMQ 5.0, retesting throughput;– Definition of message-classes for gridview-gridftp parsing records; – Testing different broker configurations: Master-Slave, JDBC persistence, file

persistence;– Testing of JMS Java clients (producers and consumers) together with python.

• 3rd Trimester(Apr/):– Network of Brokers testing;– GWT (Google Web Toolkit) Client prototype. – Setting up publishers/consumers for gridview-gridftp

• First 4 machines, now extended for additional 27 machines;

– Definition of message-classes for downtimes with OSG and initial protyping.– Moving to ActiveMQ 5.1

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

ActiveMQ updates

• Move from 4.1 => 5.0– Producer Flow Control!– AMQ Message Store, faster persistence– Command Agent– (Blob Messages)

• Now moving from 5.0 => 5.1– More stable– Default AMQ Message Store + bugfixes

• Issues with recovering from failure: very long time to replay all kept logs!

• No viable Master/Slave with fast persistence otherwise

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

MSG publish simple

• Simple python publisher script, no other dependency besides… python!

• Publishes messages over either stomp or http, depending on network availability.

• Currently version 0.9.3-1:– A few bug fixes since initial version– Priority to stomp over http.– Configurable logging

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

MSG consume2oracle

• Python consumer script– again no other dependency besides… python (and

cx_oracle)!

• How does it work:– Creates a durable subscription to the topics– Receives the messages– Retrieves the records sent along with a message– inserts them into an oracle view

• Currently version 0.9.1-1:– Bug fixes– Configurable logging

• Snapshot: – Implementing Status summaries topic (publishes summary

information back into MSG)

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Current Deployments

• OSG:– Broker at gridmsg001

• activemq 5.0.0• Persistence with journaledJDBC (derby)

– msgConsume2Oracle at gridmsg001• Publishing to SAM metricOutput/metricDescription

– msgPublishSimple• Several publishing from indiana.edu.

• Gridview:– Broker at gridmsg002

• activemq 5.0.0• Persistence with journaledJDBC (derby)

– msgPublishSimple• Called from gridview-publisher• Deployed on 27 gridftp nodes for cms t1transfers.

– msgConsume2Oracle• gridmsg002: publishing to testDatabase• Gridview machine publishing to gridview DB.

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Current Deployments

• Testing :– Brokers:

• lxb6118 / 6117 -> 5.1 Snapshot• Many different configurations along the time

(Master/Slave Network of Brokers, file persistence, etc)

– Publishers:• Java JMS producer (pcitgd24)• lxplus based msg-publish-simple

– Consumers:• msg-consume2oracle (lxb6118/6117) • GWT test (pcitgd24)

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Next Steps

• OSG scheduled downtimes– Message Class defined– Some test messages into a test DB done– Waiting Arvind’s prototype (very soon)

• JobDetails for Dirac3 detailed logging– Interested in getting information such as

cputime, and considering using MSG to do it.– Message class under definition

• MSG System Management:– How to operate a messaging system in

production?– We will need to know what’s going on!

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Thank you for your attention.

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Support Slides

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Some recent test results

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Some recent test results

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Some recent test results

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Some recent test results

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Console, GridMSG001

Broker restarts: 10 Mar.18 Apr, too many open connections -> increase of file descriptors.Since 18 April, 220000 messages.

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

InternetServices

Console, GridMSG002

420087 messages forwarded to each consumer (test + GV).