Download - DevoxxFR 2016 - 3 degrees of MoM
![Page 1: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/1.jpg)
#DevoxxFR
3 degrees of MoM kafka, rabbitmq, nsq
Guillaume Arnaud @guillarnaud
voyages-sncf.com
1
![Page 2: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/2.jpg)
#DevoxxFR 2
P publisher
C consumer
A Message-oriented Middleware (MoM)just a queue ?
![Page 3: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/3.jpg)
#DevoxxFR 3
ActiveMQ RabbitMQ Nats NSQ ZeroMQ Kafka AmazonSQS Disque Aeron Mosquitto
Beanstalk Apollo Nanomsg …
JMS Stomp MQTT AMQP XMPP …
Flume Redis Logstash Syslog …
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/
![Page 4: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/4.jpg)
#DevoxxFR 4
ActiveMQ RabbitMQNats NSQZeroMQ KafkaAmazonSQS Disque Aeron Mosquitto
Beanstalk Apollo Nanomsg …
JMS Stomp MQTT AMQP XMPP …
Flume Redis Logstash Syslog …
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaginghttp://queues.io/
ROUTING
DECENTRALIZED
DISTRIBUTED
![Page 5: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/5.jpg)
#DevoxxFR
RabbitMQ.id
5
parents: Pivotal age: v3.6.1 (march 2016) license: mozilla public license native languages: erlang/amqp fluently spoken: python, java, ruby, .Net, nodejs big brother: activemq
![Page 6: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/6.jpg)
#DevoxxFR 6
P
C
decouplingexchange queue routing/binding keys
C
exchange
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
routing
queue: devoxx2016_service
![Page 7: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/7.jpg)
#DevoxxFR 7
P
C C
exchange: fanout
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
message 1
fanoutbroadcast no routing rules
message 1 message 1
![Page 8: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/8.jpg)
#DevoxxFR 8
P
C C
exchange: direct
routing key: devoxx.2016.newuser
binding key: devoxx.2016.newuser
message 1
directrouting key=binding key
message 1
queue: new_user_servicequeue: update_user_service
binding key: devoxx.2016.updateuser
![Page 9: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/9.jpg)
#DevoxxFR 9
P
C C
exchange: topic
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
message 1
message 1 message 1
queue: devoxx_servicequeue: trigger_new_user
binding key: devoxx.2016.newuser
topicwildcard subscription
![Page 10: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/10.jpg)
#DevoxxFR
RabbitMQ.model
10
exchange routing key
queue service
event type
binding key aggregator
business domain
![Page 11: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/11.jpg)
#DevoxxFR
RabbitMQ.microservices
11
exchange'user' / fanout
cv.*
microservicesrouting between exchanges and services
specify_user_action
audit
check_fraud
exchange'cv' / topic
alert_recruiters
crop_photo
cv.photo
cv.employer.new
check_employer
exchange'mailer' / direct
generate_and_send_mail
![Page 12: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/12.jpg)
#DevoxxFR
RabbitMQ.ops
12
• virtual hosts / security • plugins: management, ldap, mqtt, consistent hashing • cluster/federation • queue size/disk • cpu
![Page 13: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/13.jpg)
#DevoxxFR
RabbitMQ.admin
13
$ rabbitmq-plugins enable rabbitmq-management
![Page 14: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/14.jpg)
#DevoxxFR
Kafka.id
14
parents: LinkedIn/confluent age: v0.9.0.1 (feb. 2016) license: apache native language: java/scala family: hdfs, spark streaming, storm, flume
![Page 15: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/15.jpg)
#DevoxxFR 15
P publisher
C consumer
5 4 3 2 1
offset
lag
log file(s)append only zero copy ~ k*100 000 msg/s
![Page 16: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/16.jpg)
#DevoxxFR 16
P
C
5 4 3 2 1
Partitionsplit the topic partition key
4 3 2 1
keys 1, 3, 5… keys 2, 4, 6…
![Page 17: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/17.jpg)
#DevoxxFR 17
P
C
distribution
5 4 3 2 1
4 3 2 1
4 3 2 1
4 3 2 1
node 1 node 2
![Page 18: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/18.jpg)
#DevoxxFR 18
replication
5 4 3 2 1
4 3 2 1
4 3 2 1
P
replica lag
partition leader
partition follower
partition follower
![Page 19: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/19.jpg)
#DevoxxFR 19
P
C
5 4 3 2 1
4 3 2 1
C
Scalingone consumer per partition add partitions for scaling
![Page 20: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/20.jpg)
#DevoxxFR 20
P
C
5 4 3 2 1
4 3 2 1
C
Scalingone consumer per partition add partitions for scaling
C
![Page 21: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/21.jpg)
#DevoxxFR 21
P
C C
Scalingone consumer per partition add partitions for scaling
C
![Page 22: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/22.jpg)
#DevoxxFR 22
P
C C
Groupconsume same messages ~ JMS topic C
group 1
group 2
![Page 23: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/23.jpg)
#DevoxxFR
Kafka.event sourcing
23
domain object
domain object
eventview 1
view 2event store
![Page 24: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/24.jpg)
#DevoxxFR
Kafka.event sourcing
24
1 id:1234 my order (…)
2 id:abcd an other order
3 id:wxyz and again…
120 id:1234 my other order
…
log compaction
offset key payload
![Page 25: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/25.jpg)
#DevoxxFR
Kafka.ops
25
• clients are part of the distributed system (heartbeat, tcp config, etc…)
• lag sender/consumer • replica lags • preferred replica election • repartition master/slave • don't forget Zookeeper • size and time retention • use kafka tools
~ equivalent to a NoSQL database
properties #
server 127
publisher 47
consumer 45
![Page 26: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/26.jpg)
#DevoxxFR
Kafka.admin
26
https://github.com/yahoo/kafka-manager
![Page 27: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/27.jpg)
#DevoxxFR
NSQ.id
27
parents: nsq.io age: v0.3.7 (feb. 2016) license: open source native language: go fluently spoken: go, python, javascript, http(!) zodiacal sign: syslog++
![Page 28: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/28.jpg)
#DevoxxFR 28
P
C
topic: server_start_eventnsqd
channel: server_subscribe
topic: server_start_eventnsqd
channel: server_subscribe
P
consumer: server_subscribe
Cnsqlookupd
http http
tcptcp
![Page 29: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/29.jpg)
#DevoxxFR 29
server
nsqd
server
nsqd
server
nsqd
server
nsqd
push
pull
decentralizedlocal push remote pull
![Page 30: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/30.jpg)
#DevoxxFR 30
nsqd
nsqd
nsqd
haproxy slave
daemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
7
use caseservice discovery
nsqd
tomcat
![Page 31: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/31.jpg)
#DevoxxFR
NSQ.service discovery
31
nsqd
nsqd
nsqd
haproxy slave
daemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
7
nsqd
tomcat
![Page 32: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/32.jpg)
#DevoxxFR
NSQ.ops
32
$ tree bin/ bin/ ├── nsq_pubsub ├── nsq_stat ├── nsq_tail ├── nsq_to_file ├── nsq_to_http ├── nsq_to_nsq ├── nsqadmin ├── nsqd ├── nsqlookupd └── to_nsq
![Page 33: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/33.jpg)
#DevoxxFR
NSQ.ops
33
$ nsqd —help -e2e-processing-latency-percentile value … -max-bytes-per-file int -max-heartbeat-interval duration -max-msg-size int -max-msg-timeout duration -max-output-buffer-size int -max-output-buffer-timeout duration -max-rdy-count int -max-req-timeout duration -mem-queue-size int -msg-timeout string -snappy …
![Page 34: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/34.jpg)
#DevoxxFR
MoM.diversity
34
![Page 35: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/35.jpg)
#DevoxxFR
NSQ.admin
35
http://nsq.io/components/nsqadmin.html
![Page 36: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/36.jpg)
#DevoxxFR
your MoM is a part of your application
36
![Page 37: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/37.jpg)
#DevoxxFR
your application is a part of your distributed system
37
![Page 38: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/38.jpg)
#DevoxxFR 38
http://www.meetup.com/fr-FR/Meetup-Messaging-Paris/
![Page 39: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/39.jpg)
#DevoxxFR
Questions ?
39
![Page 40: DevoxxFR 2016 - 3 degrees of MoM](https://reader034.vdocuments.net/reader034/viewer/2022042706/5878940b1a28ab375f8b638f/html5/thumbnails/40.jpg)
#DevoxxFR
Thanks
40