IT Messaging Services

Lionel Cons, Luca Magnoni – CERN IT/CM

28 Jan 2016 IT Messaging Services 1

Messaging

28 Jan 2016 IT Messaging Services 2

10,000 Feet Overview

28 Jan 2016 IT Messaging Services 3

Messaging Concepts

JMS defines some key concepts:

• message as body, header and properties

☞ think data and meta-data

• topic aka publish/subscribe model

☞ similar to television broadcasting

• queue aka point-to-point model

☞ similar to electronic mail

Other models exist such as virtual queues.

28 Jan 2016 IT Messaging Services 4

Messaging Protocols

There are several messaging protocols:

• AMQP (Advanced Message Queuing Protocol)

• MQTT (Message Queuing Telemetry Transport)

• OpenWire

• STOMP (Simple Text Oriented Messaging Protocol)

• XMPP (Extensible Messaging and Presence Protocol)

• ØMQ aka ZeroMQ

28 Jan 2016 IT Messaging Services 5

Examples with STOMP

28 Jan 2016 IT Messaging Services 6

→ CONNECT

login: ...

passcode: ...

← CONNECTED

→ SEND

destination: ...

some-header: ...

... body ...

→ CONNECT

...

← CONNECTED

→ SUBSCRIBE

destination: ...

← MESSAGE

...

← MESSAGE

...

What about Flume?

28 Jan 2016 IT Messaging Services 7

SinkChannelSource

• Source can be jms (inc. with STOMP provider)

• Flume can be a message consumer

• Flume is not a message broker

• no on-demand subscriptions

What about Logstash?

28 Jan 2016 IT Messaging Services 8

OutputInput

• Input can be stomp (or rabbitmq, xmpp, zeromq)

• Output can be stomp too (or ...)

• Logstash can be a message consumer and/or a message producer

• Logstash is not a message broker• no on-demand subscriptions

Filter

What about Kafka?

28 Jan 2016 IT Messaging Services 9

“Apache Kafka is publish-subscribe messaging

rethought as a distributed commit log.”

Key Kafka concepts:

• commit log, can be partitioned and replicated

• consumer group, allows JMS semantics:

• topic: all consumers use different groups

• queue: all consumers use the same group

• retention, i.e. when messages get deleted

Kafka Retention

28 Jan 2016 IT Messaging Services 10

Stream 1

Stream 2

available messages

Kafka Protocol

28 Jan 2016 IT Messaging Services 11

Kafka uses its own protocol that puts more

burden on the consumers that need to keep

track of where they are in the commit logs.

OTOH, they can read again older messages.

Kafka can be seen as a kind of message

broker with non-standard protocol and features.

Kafka’s maturity needs to be assessed. The

new consumer API is marked “beta quality”.

Messaging Software in 2012

28 Jan 2016 IT Messaging Services 12

Broker Qpid MRG HornetQ ActiveMQ Apollo RabbitMQ

Language Java C++ Java Java Scala Erlang

Main Protocols

AMQP AMQP proprietarySTOMP

OpenWireSTOMPAMQPMQTT

OpenWireSTOMPAMQPMQTT

AMQPSTOMP(MQTT)

Owner (*) Red Hat FuseSource(Progress)

VMware

September 7, 2012 : Red Hat completed its acquisition of FuseSource

Red Hat

IT Messaging Services

28 Jan 2016 IT Messaging Services 13

Hardware Layer

We currently use physical machines for the

brokers (and VMs for the monitoring nodes).

New types of VMs should be tried:

• high-end VMs with fast Ceph volumes

• high-end VMs with dedicated local SSD drives

To be tested and benchmarked…

28 Jan 2016 IT Messaging Services 14

Operating System Layer

We currently use Java 7 on SL 6.

We need to prepare for Java 8 and CentOS 7.

History taught us that there is no such thing as

a painless upgrade.

28 Jan 2016 IT Messaging Services 15

Software Layer

We currently use Red Hat’s A-MQ which is

basically Apache ActiveMQ patched, tested

and supported by Red Hat.

We currently use A-MQ 6.2.1 which is based

on ActiveMQ 5.11.

Since our clients only use STOMP, we could

migrate to other software such as RabbitMQ

or Apache ActiveMQ Artemis.

28 Jan 2016 IT Messaging Services 16

Messaging Services Growth

One year ago:

• 5 (+ 5) clusters: Agile Infrastructure, ATLAS,

Dashboard, IT/DSS and MIG

• 23 brokers on 16 machines

Today:

• 8 (+ 7) clusters with the addition of: IT/DB,

MCollective and Network Monitoring (perfSONAR)

• 32 brokers on 16 machines

28 Jan 2016 IT Messaging Services 17

Messaging Services Numbers

28 Jan 2016 IT Messaging Services 18

Messaging Services Users

Many different use cases. For instance:

• AI: events, monitoring notifications

• ATLAS: Rucio, event indexes

• Dashboard: FTS, Transfer, SAM, XRootD

• DSS: CASTOR/EOS logs

• MCollective

• MIG: monitoring

• Network Monitoring: perfSONAR

28 Jan 2016 IT Messaging Services 19

AI Monitoring Use Case

Very low volume (only notifications) but very

diverse clients: PES, River, SNow, Spectrum...

Many clients: any machine using AI monitoring.

Enabling X.509 authentication:

• 50k+ DNs authorized

• pushed ActiveMQ’s limits thanks to Red Hat

• we could go even further by submitting a new,

more flexible X.509 authentication module

28 Jan 2016 IT Messaging Services 20

MCollective Use Case

MCollective needs a “network of brokers”.

Requirements are challenging:

• 30k concurrent connections (vs. 3k in total before)

• 300k subscriptions (vs. 1k in total before)

• working with Red Hat for the design and tuning

First prototype delivered and is being tested.

Several MCollective shortcomings found and

reported.

28 Jan 2016 IT Messaging Services 21

Messaging Team

Currently: ~1 FTE across two persons.

Wide range of activities:

• operations and evolutions (e.g. CC7)

• support (with the help of Red Hat for A-MQ)

• consultancy

• testing and validation

• software maintenance and development

Contact: http://cern.ch/mig – mig@cern.ch

28 Jan 2016 IT Messaging Services 22

Messaging Software

Diverse software:

• libraries: STOMP for Perl, messaging for Python...

• building blocks: directory queue, stompclt...

• management: MBSD, MBCG, Puppet hostgroup...

• monitoring: MBM, Metis...

• testing: MBTF...

In total:

• ~70k SLOC mainly in Perl, Python, Java and C

• 14 packages in EPEL + ~18 in CERN’s Koji

28 Jan 2016 IT Messaging Services 23

Metis

We are very happy with our Esper based

solution named Metis.

For more information, see our “Advanced

Monitoring With Complex Stream Processing”

presentation at the IT Technical Forum.

Most of Metis is generic and should be reused

in other areas where advanced monitoring is

needed.

28 Jan 2016 IT Messaging Services 24

Possible Next Steps

1. Investigate Kafka-based messaging services

2. Improve and extend our building blocks

3. Test high-end VMs with better I/O:

• fast Ceph volumes

• local SSD drives

4. Extend Metis beyond messaging monitoring

5. Evaluate ActiveMQ Artemis

6. Prepare for the upcoming system upgrade

• CentOS 7 and Java 8

28 Jan 2016 IT Messaging Services 25