queues queues queues — how rabbitmq enables reactive architectures

Post on 07-Jan-2017

459 Views

Category:

Software

3 Downloads

Preview:

Click to see full reader

TRANSCRIPT

Martin TajurCTO, Co-Founder

February 16, 2016DevClub XLIII meetup, Tallinn, Estonia

Hello.

● Web-based CRM for small teams with big ambitions.

● Web-based CRM for small teams with big ambitions.● Founded in 2010.

● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.

● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.● 140+ employees, venture funded (BVP, Series A in 2015)

● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.● 140+ employees, venture funded (BVP, Series A in 2015)● Engineering, Product, UX, Marketing in Tallinn and Tartu.

Marketing, BizDev in New York, NY.

● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.● 140+ employees, venture funded (BVP, Series A in 2015)● Engineering, Product, UX, Marketing in Tallinn and Tartu.

Marketing, BizDev in New York, NY.● Very end user focused, helping the actual sales person do

their job.

● 20,000+ simultaneous online users

● 20,000+ simultaneous online users

● 800+ API req/sec

● 20,000+ simultaneous online users

● 800+ API req/sec

● 400,000+ incoming emails per day

● 20,000+ simultaneous online users

● 800+ API req/sec

● 400,000+ incoming emails per day

● Started with Node.js based microservices, reactive

architecture in 2012

● 20,000+ simultaneous online users

● 800+ API req/sec

● 400,000+ incoming emails per day

● Started with Node.js based microservices, reactive

architecture in 2012

● In production with first Docker based services

● 20,000+ simultaneous online users

● 800+ API req/sec

● 400,000+ incoming emails per day

● Started with Node.js based microservices, reactive

architecture in 2012

● In production with first Docker based services

● In total, 500+ VMs/hosts/instances

Queues, queues, queuesHow RabbitMQ enables reactive architectures

Martin TajurCTO, Co-Founder

February 16, 2016DevClub XLIII meetup, Tallinn, Estonia

➔ Co-Founder and CTO of Pipedrive.

➔ Started career in 2001 as a designer, later a full stack dev.

➔ Part of ex-Skype mafia

About me

Reactive architectures

The Reactive Manifesto

Manifestos are good when

➔ they are used as building blocks

➔ they add, rather than compete or subtract

➔ they tap into the growing wisdom, and extend it

Ok, so,the reactive manifesto...

An initiative to define atech agnostic shared vocabulary ofarchitectural patterns for reactive systems.

It states that...

Systems should be reactive to

Systems should be reactive to

Events

Systems should be reactive to

Events Load

Systems should be reactive to

Events Load

Failure

Systems should be reactive to

Events Load

Failure Users

Systems should be reactive to tolerate

Events Load

Failure Users

ResponsiveReact to users in

timely manner

http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385

Goal

ResponsiveReact to users in

timely manner

ResilientReact to failures

ElasticReact to load

http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385

Goal

Principles

ResponsiveReact to users in

timely manner

ResilientReact to failures

ElasticReact to load

Message-drivenComponent-to-component interaction

http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385

Goal

Principles

Method

Systems built as Reactive Systems are more flexible, loosely-coupled and scalable.

This makes themeasier to develop and amenable to change.

They are significantly more tolerant of failure and when failure does occur they meet it with elegance rather than disaster.

Reactive Systems arehighly responsive, giving users effective interactive feedback.

How queues fit in to the picture?

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

http://www.iron.io/top-10-uses-for-message-queue/

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Decoupling

➔ Hard to predict future needs at the start of a project.

➔ Message queues create an implicit, data-based interface that different services can implement.

➔ Allows you to extend and modify processes independently, ensuring they adhere to the same interfaces.

➔ Lets you swap, mix and add queue consuming services.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Redundancy

➔ Sometimes, services die when processing data.

➔ Unless that data is persisted, it’s lost forever.

➔ Queues mitigate this by persisting data until it has been fully processed.

➔ No job gets lost.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Load Balancing, Scalability

➔ It’s easy to scale up the rate with which messages are added to the queue or processed – by adding more instances.

➔ Easy to create load balancing by attaching multiple workers to a single queue.

➔ No code changes, no configurations need to be tweaked.

➔ Scaling is as simple as adding more power.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Elasticity & Spikability

➔ When your application hits the front page of Hacker News, you’re going to see unusual levels of traffic.

➔ Your application needs to be able to keep functioning.

➔ But the traffic is anomaly, not the standard — it’s wasteful to have enough resources on standby to handle these spikes.

➔ Message queues will allow components to struggle through the increased load, instead of getting overloaded with requests and failing completely.

➔ Queue lengths and consumer utilization = basis for auto-scaling.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Resiliency

➔ When part of your architecture fails, it doesn’t need to take the entire system down :-)

➔ Message queues decouple services, so if a service that is processing messages from the queue fails, messages can still be added to the queue to be processed when the system recovers.

➔ This ability to accept requests that will be retried or processed at a later date is often the difference between an inconvenienced customer and a frustrated customer.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Delivery guarantees

➔ The redundancy provided by message queues guarantees that a message will eventually be processed.

➔ No matter how many processes are pulling data from the queue, each message will be processed at least once.

➔ This is often made possible using a way to “reserve” messages being processed, temporarily removing them from the queue.

➔ Unless the client specifically states that it’s finished with that message, the message will be placed back to the top of the queue.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Ordering Guarantees

➔ In a lot of situations, the order with which data is processed is important.

➔ Message queues are inherently ordered, and capable of providing guarantees that data will be processed in a specific order.

➔ Most message queues use FIFO (first in, first out), so the order in which messages are placed on a queue is the order in which they’ll be retrieved from it.

➔ (Beware when using multiple consumers)

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Buffering

➔ In any non-trivial system, there are going to be components that require different processing times.

➔ For example, it takes less time to upload an image than it does to apply a filter to it.

➔ You can also collect a certain number of items together and process them as a single batch.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Data Flow Visibility

➔ In a distributed system, getting an overall sense of where and how data flows can be a daunting task.

➔ Message queues, and routing rules, help identify and understand data flow paths.

➔ Through the rate with which they are processed, one can easily identify under-performing processes or areas where the data flow is not optimal.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

Async Communication

➔ A lot of times, you don’t want to or need to process a message immediately.

➔ Message queues enable asynchronous processing, which allows you to put a message on the queue without processing it immediately.

➔ Queue up as many messages as you like, then process them at your leisure.

➔ Opens possibilities for retry-later setups.

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

10 benefits of using message

queues

Decoupling

Redundancy

Load Balancing, Scalability

Elasticity & Spikability

Resiliency

Delivery Guarantees

Ordering Guarantees

Buffering

Data Flow visibility

Asynchronous communication

http://www.iron.io/top-10-uses-for-message-queue/

RabbitMQ

RabbitMQ

➔ An AMQP messaging broker

➔ Written in Erlang

➔ Launched in 2007

➔ Originally written by Rabbit Technologies Ltd. in London, UK

➔ Got acquired in 2010 by a division of VMWare

➔ Spin-off in 2013 to Pivotal Software, Inc

➔ Core participant of the AMQP working group

AMQP

➔ Advanced Message Queue Protocol

➔ An open standard, independent from RabbitMQ

➔ Provider agnostic, in theory.A client should be able to swap RabbitMQ with different AMQP server

➔ Client libraries available for all major programming languages

➔ Most recent specification version is 1.0. *

* RabbitMQ mostly AMQP version 0.9.1 as of Feb 2016.

RabbitMQ server

Basic concept

binding

Publisher

RabbitMQ server

Basic concept

Publisher Exchange

RabbitMQ server

Basic concept

Publisher

Queue

Queue

Exchange

RabbitMQ server

Basic concept

Publisher

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Basic concept

Publisher

ConsumerQueue

Queue

binding

binding

Exchange

RabbitMQ server

Basic concept

Consumer

Consumer

Consumer

Publisher

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Basic concept

statisticsqueue

webhookqueue

#

#.company_15

dealsexchange

Deals service

statistics service

webhook service

webhook service

Queue

Queue

binding

binding

Exchange

Consumer

Consumer

Consumer

RabbitMQ server

Basic concept

Publisher

Publishing

Publishing

➔ Your apps publish messages. Each message has

◆ routing key (e.g. subject)

◆ delivery mode (persistent or not persistent)

◆ headers (e.g. { "content-type": "application/json" })

◆ payload (e.g. { "some": "json" })

◆ properties (e.g. correlation-id)

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Consumer

Consumer

Consumer

Basic concept

Publisher

Publishing

RabbitMQ server

Exchange

Basic concept

Consumer

Consumer

Consumer

Publisher

Queue

Queue

binding

binding

Exchange

Exchanges

➔ Exchanges are AMQP entities where messages are sent to.

➔ Exchanges on their own are useless,unless bound to something.

https://www.rabbitmq.com/tutorials/amqp-concepts.html

Four types of exchanges

➔ Direct, Fanout, Topic, Headers

➔ Type defines how you can bind from it

➔ Most useful is Topic (allows regexp-like bindings)

➔ Read more from RabbitMQ docs

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Exchange

Basic concept

Consumer

Consumer

Consumer

Publisher

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Basic concept

Bindings

Consumer

Consumer

Consumer

Publisher

Bindings

➔ Rules with which messages are routedfrom exchanges to queues

➔ Each queue and exchange can have multiple bindings

➔ Examples: [characteristic].[color].[kind]◆ *.orange.*◆ *.*.rabbit◆ lazy.#

Bindings

Q1 is interested in all the orange animals.

Q2 wants to hear everything about rabbits, and everything about lazy animals.

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Basic concept

Bindings

Consumer

Consumer

Consumer

Publisher

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Basic concept

Consumer

Consumer

Consumer

Publisher

Queues

Queues

➔ Ordered lists of messages to be consumed

➔ FIFO

➔ Can be either durable or transient

➔ Can have arguments (behavioral properties)TTL of messages in itauto-expirymaximum lengthdead letter handling

Queue

Queue

binding

binding

Exchange

RabbitMQ server

Basic concept

Consumer

Consumer

Consumer

Publisher

Queues

RabbitMQ server

Basic concept

Queue

Queue

binding

binding

Exchange

Consuming

Consumer

Consumer

Consumer

Publisher

Consuming

➔ Act of receiving and acting upon messages from a queue

Consuming

➔ Act of receiving and acting upon messages from a queue

➔ One at a time, or multiple-in-flight with acking

Consuming

➔ Act of receiving and acting upon messages from a queue

➔ One at a time, or multiple-in-flight with acking

➔ Delivery and execution guarantees

Consuming

➔ Act of receiving and acting upon messages from a queue

➔ One at a time, or multiple-in-flight with acking

➔ Delivery and execution guarantees

➔ One queue can have multiple consumers (enables load balancing — but then exact order is not guaranteed)

Consuming

➔ Act of receiving and acting upon messages from a queue

➔ One at a time, or multiple-in-flight with acking

➔ Delivery and execution guarantees

➔ One queue can have multiple consumers (enables load balancing — but then exact order is not guaranteed)

➔ A consumer may ask for exclusivity

Consumption sequence diagram

RabbitMQ Consumer

RabbitMQ Consumer

[message]

Consumption sequence diagram

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

Consumption sequence diagram

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’ attempt to process

the message

Consumption sequence diagram

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’ attempt to process

the messageack

Consumption sequence diagram

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’ attempt to process

the messageack

mark message delivered

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

attempt to process the message

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

attempt to process the message

dies!

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

attempt to process the message

(connection reset) dies!

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

attempt to process the message

(connection reset) dies!

put message back to ‘ready’

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

attempt to process the message

(connection reset) dies!

put message back to ‘ready’

Consumer

Consumer

Consumption sequence diagram with a failure

RabbitMQ Consumer

RabbitMQ Consumer

[message]mark

message ‘unacked’

attempt to process the message

(connection reset) dies!

put message back to ‘ready’

Consumer

Consumer

[message]mark message ‘unacked’

Consumer characteristics (Node.js)

➔ Persistent TCP connection to AMQP server

➔ RabbitMQ pushes messages to consumer (no polling)

➔ Connection is multiplexed (using channels)

➔ Event listener/callback function is executed per each message, with ack() callback provided to be called when a message can be considered consumed.

RabbitMQ server

Basic concept

Publisher

Queue

Queue

binding

binding

Consumer

Consumer

Consumer

Exchange

RabbitMQ server

Basic concept

Deals service

statisticsqueue

webhookqueue

#

#.company_15

statistics service

webhook service

dealsexchange

webhook service

RabbitMQ server

Basic concept

Deals service

dealsexchange

Photos service

statisticsqueue

webhookqueue

statistics service

webhook service

webhook service

photosexchange

RabbitMQ server

Basic concept

Deals service

statisticsqueue

webhookqueue

statistics service

webhook service

dealsexchange

webhook service

Photos service

photosexchange

analyzerqueue

analyzer service

RabbitMQ server

Basic concept

Deals service

statisticsqueue

webhookqueue

statistics service

webhook service

dealsexchange

webhook service

Photos service

photosexchange

analyzerqueue

analyzer service

resizingqueue photo resizer

service

So it’s like…a series of tubes.

Some like to call it“a post box, a post office and a postman, all in one”...

“When you send mail to the post box you're pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman.”From the RabbitMQ official tutorial.

However,in a microservices environment, this post metaphor can be dangerous.

The post metaphor can lure you into publishing messages with a specific consumer in mind,

thus increasing tight coupling which makes it harder to add other kinds of consumers in the future without having to change multiple services.

Example:Orchestration vs choreography

User signs up

Example use case shown by Sam Newman in “Principles of microservices” presentation

User signs up

Create user account

Example use case shown by Sam Newman in “Principles of microservices” presentation

User signs up

Create user account

Sign up to newsletter at MailChimp

Example use case shown by Sam Newman in “Principles of microservices” presentation

User signs up

Create user account

Sign up to newsletter at MailChimp

Send welcome email

Example use case shown by Sam Newman in “Principles of microservices” presentation

User signs up

Create user account

Sign up to newsletter at MailChimp

Send welcome email Send welcome gift

Example use case shown by Sam Newman in “Principles of microservices” presentation

User signs up

Create user account

Sign up to newsletter at MailChimp

Send welcome email Send welcome gift

Example use case shown by Sam Newman in “Principles of microservices” presentation

Done!

Orchestration

Example use case shown by Sam Newman in “Principles of microservices” presentation

Sign up service

Orchestration

Example use case shown by Sam Newman in “Principles of microservices” presentation

Orchestration

Sign up service

Newsletter service

subscribe

Example use case shown by Sam Newman in “Principles of microservices” presentation

Orchestration

Sign up service

Newsletter service

Email sendersubscribe

send email

Example use case shown by Sam Newman in “Principles of microservices” presentation

Orchestration

Sign up service

Newsletter service

Email sender

Delivery service

subscribe

send email

send package

➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind.

Problem with orchestration — tight coupling

➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind.

➔ Makes it hard to plug in new services along the way, and thus shape the system organically.

Problem with orchestration — tight coupling

➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind.

➔ Makes it hard to plug in new services along the way, and thus shape the system organically.

➔ Expectations of replies to completed commands — seeing a “send this newsletter out” command in service A may seem like it is a synchronous action, whereas with a message queue it is not.

Problem with orchestration — tight coupling

Choreography

https://www.flickr.com/photos/stevenpisano/16313427202

Choreography

Sign up service

Example use case shown by Sam Newman in “Principles of microservices” presentation

Example use case shown by Sam Newman in “Principles of microservices” presentation

Choreography

Sign up service

user signed up

Example use case shown by Sam Newman in “Principles of microservices” presentation

Choreography

Sign up service

Newsletter service

user signed up

Example use case shown by Sam Newman in “Principles of microservices” presentation

Choreography

Sign up service

Email sender

Newsletter service

user signed up

Example use case shown by Sam Newman in “Principles of microservices” presentation

Choreography

Sign up service

Email sender

Delivery service

Newsletter service

user signed up

Choreography

Sign up service

Newsletter service

Email sender

Delivery service

RabbitMQ

Queue

Queue

signups Queue

Example use case shown by Sam Newman in “Principles of microservices” presentation

➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.

Choreography

➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.

➔ ResilientFailure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed.

Choreography

➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.

➔ ResilientFailure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed.

➔ ElasticQueue lengths can be used to scale consuming services up or down.

Choreography

➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.

➔ ResilientFailure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed.

➔ ElasticQueue lengths can be used to scale consuming services up or down.

➔ Works well with message queues. Allows reactive architectures.

Choreography

So, embrace choreography wherever possible.

Dumb pipes,smart endpoints

Dumb service A

Dumb service B

Dumb service C

Dumb service D

Dumb service A

Dumb service B

Dumb service C

Dumb service D

Enterprise Service Bus

Dumb service A

Dumb service B

Dumb service C

Dumb service D

Dumb service A

Dumb service B

Dumb service C

Dumb service D

Magic Mystery Bus

Example credit to Sam Newman in “Principles of microservices” presentation.

Dumb service A

Dumb service B

Dumb service C

Dumb service D

Dumb service A

Dumb service B

Dumb service C

Dumb service D

Magic Mystery Bus

Dumb pipes, smart endpoints

➔ Initially, you deploy an empty RabbitMQ.

Dumb pipes, smart endpoints

➔ Initially, you deploy an empty RabbitMQ.

➔ Do not make RabbitMQ deployment be aware of the desired data flows.

Dumb pipes, smart endpoints

➔ Initially, you deploy an empty RabbitMQ.

➔ Do not make RabbitMQ deployment be aware of the desired data flows.

➔ Exchanges, bindings and queues are created by services themselves as they need.

RabbitMQ server

Basic concept

Deals service

statisticsqueue

webhookqueue

#

#.company_15

statistics service

webhook service

dealsexchange

webhook service

RabbitMQ server

Basic concept political map

Deals service

statisticsqueue

webhookqueue

#

#.company_15

statistics service

webhook service

dealsexchange

webhook service

RabbitMQ server

Basic concept political map

Deals service

statisticsqueue

webhookqueue

#

#.company_15

statistics service

webhook service

dealsexchange

webhook serviceKingdom of

Deals Service

Commonwealth of Statistics Service

Republic of Webhook Service

So, in essence, RabbitMQ is left with nothing but the dumb pipes.

Smart service A

Smart service B

Smart service C

Smart service D

Smart service A

Smart service B

Smart service C

Smart service D

Dumb pipes

RabbitMQin the wild

RabbitMQ at Pipedrive

➔ Started using in 2012

RabbitMQ at Pipedrive

➔ Started using in 2012➔ Main backbone of async service-to-service communication

RabbitMQ at Pipedrive

➔ Started using in 2012➔ Main backbone of async service-to-service communication➔ All data change events are published to RabbitMQ.

RabbitMQ at Pipedrive

➔ Started using in 2012➔ Main backbone of async service-to-service communication➔ All data change events are published to RabbitMQ.➔ Averaging around >1,700 msg/sec. Peaking sometimes at

3,000+ msg/sec. >140M msg/day.

RabbitMQ at Pipedrive

➔ Started using in 2012➔ Main backbone of async service-to-service communication➔ All data change events are published to RabbitMQ.➔ Averaging around >1,700 msg/sec. Peaking sometimes at

3,000+ msg/sec. >140M msg/day.

RabbitMQ at Pipedrive

RabbitMQ at Pipedrive

➔ Initially deployed a 3-node cluster

MQ

MQ

MQ

RabbitMQ at Pipedrive

➔ Initially deployed a 3-node cluster➔ Then scaled it up to a 5-node cluster

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

RabbitMQ at Pipedrive

➔ Initially deployed a 3-node cluster➔ Then scaled it up to a 5-node cluster➔ Then decided to move to multiple 3-node clusters instead

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

MQ

RabbitMQ at Pipedrive

RabbitMQ at Pipedrive

➔ Started seeing networking overhead in a 5-node cluster as the load grew.

RabbitMQ at Pipedrive

➔ Started seeing networking overhead in a 5-node cluster as the load grew.

➔ Multiple clusters with semi-isolated traffic (bound by either SLA or logical service groups) was easier to handle for us.

RabbitMQ at Pipedrive

➔ Started seeing networking overhead in a 5-node cluster as the load grew.

➔ Multiple clusters with semi-isolated traffic (bound by either SLA or logical service groups) was easier to handle for us.

➔ It was a logical architecture decision, not a technical constraint — RabbitMQ does scale well beyond our load.

3 caveats

1.Who owns an exchange?

Exchange ownership question

Exchange ownership question

➔ Ultimately, every service will depend on existence of certain exchanges.

Exchange ownership question

➔ Ultimately, every service will depend on existence of certain exchanges.

➔ If these are missing, binding will throw an error.

Exchange ownership question

➔ Ultimately, every service will depend on existence of certain exchanges.

➔ If these are missing, binding will throw an error.

➔ But which service should create the exchange?There should ideally be only one owner per each exchange.

Exchange ownership question

➔ Ultimately, every service will depend on existence of certain exchanges.

➔ If these are missing, binding will throw an error.

➔ But which service should create the exchange?There should ideally be only one owner per each exchange.

➔ At Pipedrive, we have so far had shared exchange ownerships. Eventually it could create problems down the line as exchange properties are defined in multiple services.

Exchange ownership question

➔ Ultimately, every service will depend on existence of certain exchanges.

➔ If these are missing, binding will throw an error.

➔ But which service should create the exchange?There should ideally be only one owner per each exchange.

➔ At Pipedrive, we have so far had shared exchange ownerships. Eventually it could create problems down the line as exchange properties are defined in multiple services.

➔ Possible solution: die consumers when an exchange does not exist?

2.Adding/removing bindings is like changing a database schema

Adding/removing bindings is like changing schema

Adding/removing bindings is like changing schema

➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.

Adding/removing bindings is like changing schema

➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.

➔ It takes time.

Adding/removing bindings is like changing schema

➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.

➔ It takes time.

➔ At Pipedrive, we used to create a queue+binding for each logged in user to facilitate websocket connection back to end user’s browser.

Adding/removing bindings is like changing schema

➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.

➔ It takes time.

➔ At Pipedrive, we used to create a queue+binding for each logged in user to facilitate websocket connection back to end user’s browser.

➔ Don’t do it.

tl;dr:RabbitMQ is optimized for throughput of messages, not for throughput of schema changes on the fly.

At Pipedrive, we have a microservice we call socketqueue.

It’s a piping service betweenuser’s browser and RabbitMQ

websocket User’s browser

socketqueue(consumer)

RabbitMQ

REST APIs(publishers)

AMQPAMQP

Socketqueue 1.0 architecture

➔ One queue + binding per each connected user

➔ One Websocket connection per each connected user

➔ One-to-one relation between queue and websocket inside the socketqueue service.

➔ Horizontally scalable

RabbitMQ cluster

Socketqueue 1.0 architecture

REST APIs

q3

api.eventsexchange

q2

q1

socketq1

u1u2u3

q6

q5

q4

socketq2

u4u5u6

q8 socketq3

u7u8u9

q7

+ 10K more

q9

300+ msg/sec

RabbitMQ cluster

Socketqueue 1.0 architecture

REST APIs

q3

api.eventsexchange

q2

q1

socketq1

u1u2u3

q6

q5

q4

socketq2

u4u5u6

q8 socketq3

u7u8u9

q7

+ 10K more

q9

Did not scale beyond 10K queues, bindings

300+ msg/sec

RabbitMQ cluster

Socketqueue 1.0 architecture

REST APIs

q3

api.eventsexchange

q2

q1

socketq1

u1u2u3

q6

q5

q4

socketq2

u4u5u6

q8 socketq3

u7u8u9

q7

+ 10K more

q9

300+ msg/sec

Socketqueue 2.0 architecture

REST APIs api.eventsexchange

q-socket1 socketq1

u1u2u3

q-socket2 socketq2

u4u5u6

socketq3

u7u8u9

q-socket3

RabbitMQ cluster

300+ msg/sec

Socketqueue 1.0 architecture

➔ One queue + binding per each connected user

➔ One Websocket connection per each connected user

➔ One-to-one relation between queue and websocket inside the socketqueue service.

➔ Horizontally scalable

Socketqueue 1.02.0 architecture

➔ One queue + binding per each connected user socketqueue service instance.

➔ One Websocket connection per each connected user

➔ One-to-onemany relation between queue and websocket inside the socketqueue service.

➔ Horizontally scalable well beyond 10K simultaneous users

Most problematic?

Initial setup of 10K+ queues/bindings

during version upgrades, service restarts.

3.Delivery guarantees

100% delivery guarantee

=

Publisher Acknowledgements

Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ

Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk

Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message

Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue

Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue➔ A consumer picks up the message and starts processing it

Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue➔ A consumer picks up the message and starts processing it➔ RabbitMQ server dies and comes back up

Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue➔ A consumer picks up the message and starts processing it➔ RabbitMQ server dies and comes back up➔ Except, that message that was being processed? It’s not there any more.

100% delivery guarantee

=

Publisher Acknowledgements

Read more from https://www.rabbitmq.com/confirms.html

RabbitMQ elsewhere

https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine

1 millionmessages per second

1,000,000messages per second

86,400,000,000messages per day

To put this into context...

11B2014 all of SMS traffic

per day globally

http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2

17.2B

11B2014 all of SMS traffic

per day globally

2016 Apple iMessages per day globally

http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2

17.2B

11B2014 all of SMS traffic

per day globally

2016 Apple iMessages per day globally

Facebook, Messenger, and WhatsApp per day

combined

45B

http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2

86.4B

17.2B

11B2014 all of SMS traffic

per day globally

2016 Apple iMessages per day globally

That single RabbitMQ experiment

Facebook, Messenger, and WhatsApp per day

combined

45B

http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2

I think that’s pretty remarkable.

Thank you.

Martin Tajur@tajur on Twitter

With this promo code

DEVCLUBEE2 months freewww.pipedrive.com

Thank you.

Martin Tajur@tajur on Twitter

With this promo code

DEVCLUBEE2 months freewww.pipedrive.com

We’re hiring.

Questions/Answers

top related