queues queues queues — how rabbitmq enables reactive architectures
TRANSCRIPT
Martin TajurCTO, Co-Founder
February 16, 2016DevClub XLIII meetup, Tallinn, Estonia
Hello.
● Web-based CRM for small teams with big ambitions.
● Web-based CRM for small teams with big ambitions.● Founded in 2010.
● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.
● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.● 140+ employees, venture funded (BVP, Series A in 2015)
● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.● 140+ employees, venture funded (BVP, Series A in 2015)● Engineering, Product, UX, Marketing in Tallinn and Tartu.
Marketing, BizDev in New York, NY.
● Web-based CRM for small teams with big ambitions.● Founded in 2010.● Used by over 30,000 businesses worldwide.● 140+ employees, venture funded (BVP, Series A in 2015)● Engineering, Product, UX, Marketing in Tallinn and Tartu.
Marketing, BizDev in New York, NY.● Very end user focused, helping the actual sales person do
their job.
● 20,000+ simultaneous online users
● 20,000+ simultaneous online users
● 800+ API req/sec
● 20,000+ simultaneous online users
● 800+ API req/sec
● 400,000+ incoming emails per day
● 20,000+ simultaneous online users
● 800+ API req/sec
● 400,000+ incoming emails per day
● Started with Node.js based microservices, reactive
architecture in 2012
● 20,000+ simultaneous online users
● 800+ API req/sec
● 400,000+ incoming emails per day
● Started with Node.js based microservices, reactive
architecture in 2012
● In production with first Docker based services
● 20,000+ simultaneous online users
● 800+ API req/sec
● 400,000+ incoming emails per day
● Started with Node.js based microservices, reactive
architecture in 2012
● In production with first Docker based services
● In total, 500+ VMs/hosts/instances
Queues, queues, queuesHow RabbitMQ enables reactive architectures
Martin TajurCTO, Co-Founder
February 16, 2016DevClub XLIII meetup, Tallinn, Estonia
➔ Co-Founder and CTO of Pipedrive.
➔ Started career in 2001 as a designer, later a full stack dev.
➔ Part of ex-Skype mafia
About me
Reactive architectures
The Reactive Manifesto
Manifestos are good when
➔ they are used as building blocks
➔ they add, rather than compete or subtract
➔ they tap into the growing wisdom, and extend it
Ok, so,the reactive manifesto...
An initiative to define atech agnostic shared vocabulary ofarchitectural patterns for reactive systems.
It states that...
Systems should be reactive to
Systems should be reactive to
Events
Systems should be reactive to
Events Load
Systems should be reactive to
Events Load
Failure
Systems should be reactive to
Events Load
Failure Users
Systems should be reactive to tolerate
Events Load
Failure Users
ResponsiveReact to users in
timely manner
http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385
Goal
ResponsiveReact to users in
timely manner
ResilientReact to failures
ElasticReact to load
http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385
Goal
Principles
ResponsiveReact to users in
timely manner
ResilientReact to failures
ElasticReact to load
Message-drivenComponent-to-component interaction
http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385
Goal
Principles
Method
Systems built as Reactive Systems are more flexible, loosely-coupled and scalable.
This makes themeasier to develop and amenable to change.
They are significantly more tolerant of failure and when failure does occur they meet it with elegance rather than disaster.
Reactive Systems arehighly responsive, giving users effective interactive feedback.
How queues fit in to the picture?
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
http://www.iron.io/top-10-uses-for-message-queue/
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Decoupling
➔ Hard to predict future needs at the start of a project.
➔ Message queues create an implicit, data-based interface that different services can implement.
➔ Allows you to extend and modify processes independently, ensuring they adhere to the same interfaces.
➔ Lets you swap, mix and add queue consuming services.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Redundancy
➔ Sometimes, services die when processing data.
➔ Unless that data is persisted, it’s lost forever.
➔ Queues mitigate this by persisting data until it has been fully processed.
➔ No job gets lost.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Load Balancing, Scalability
➔ It’s easy to scale up the rate with which messages are added to the queue or processed – by adding more instances.
➔ Easy to create load balancing by attaching multiple workers to a single queue.
➔ No code changes, no configurations need to be tweaked.
➔ Scaling is as simple as adding more power.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Elasticity & Spikability
➔ When your application hits the front page of Hacker News, you’re going to see unusual levels of traffic.
➔ Your application needs to be able to keep functioning.
➔ But the traffic is anomaly, not the standard — it’s wasteful to have enough resources on standby to handle these spikes.
➔ Message queues will allow components to struggle through the increased load, instead of getting overloaded with requests and failing completely.
➔ Queue lengths and consumer utilization = basis for auto-scaling.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Resiliency
➔ When part of your architecture fails, it doesn’t need to take the entire system down :-)
➔ Message queues decouple services, so if a service that is processing messages from the queue fails, messages can still be added to the queue to be processed when the system recovers.
➔ This ability to accept requests that will be retried or processed at a later date is often the difference between an inconvenienced customer and a frustrated customer.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Delivery guarantees
➔ The redundancy provided by message queues guarantees that a message will eventually be processed.
➔ No matter how many processes are pulling data from the queue, each message will be processed at least once.
➔ This is often made possible using a way to “reserve” messages being processed, temporarily removing them from the queue.
➔ Unless the client specifically states that it’s finished with that message, the message will be placed back to the top of the queue.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Ordering Guarantees
➔ In a lot of situations, the order with which data is processed is important.
➔ Message queues are inherently ordered, and capable of providing guarantees that data will be processed in a specific order.
➔ Most message queues use FIFO (first in, first out), so the order in which messages are placed on a queue is the order in which they’ll be retrieved from it.
➔ (Beware when using multiple consumers)
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Buffering
➔ In any non-trivial system, there are going to be components that require different processing times.
➔ For example, it takes less time to upload an image than it does to apply a filter to it.
➔ You can also collect a certain number of items together and process them as a single batch.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Data Flow Visibility
➔ In a distributed system, getting an overall sense of where and how data flows can be a daunting task.
➔ Message queues, and routing rules, help identify and understand data flow paths.
➔ Through the rate with which they are processed, one can easily identify under-performing processes or areas where the data flow is not optimal.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
Async Communication
➔ A lot of times, you don’t want to or need to process a message immediately.
➔ Message queues enable asynchronous processing, which allows you to put a message on the queue without processing it immediately.
➔ Queue up as many messages as you like, then process them at your leisure.
➔ Opens possibilities for retry-later setups.
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
10 benefits of using message
queues
Decoupling
Redundancy
Load Balancing, Scalability
Elasticity & Spikability
Resiliency
Delivery Guarantees
Ordering Guarantees
Buffering
Data Flow visibility
Asynchronous communication
http://www.iron.io/top-10-uses-for-message-queue/
RabbitMQ
RabbitMQ
➔ An AMQP messaging broker
➔ Written in Erlang
➔ Launched in 2007
➔ Originally written by Rabbit Technologies Ltd. in London, UK
➔ Got acquired in 2010 by a division of VMWare
➔ Spin-off in 2013 to Pivotal Software, Inc
➔ Core participant of the AMQP working group
AMQP
➔ Advanced Message Queue Protocol
➔ An open standard, independent from RabbitMQ
➔ Provider agnostic, in theory.A client should be able to swap RabbitMQ with different AMQP server
➔ Client libraries available for all major programming languages
➔ Most recent specification version is 1.0. *
* RabbitMQ mostly AMQP version 0.9.1 as of Feb 2016.
RabbitMQ server
Basic concept
binding
Publisher
RabbitMQ server
Basic concept
Publisher Exchange
RabbitMQ server
Basic concept
Publisher
Queue
Queue
Exchange
RabbitMQ server
Basic concept
Publisher
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Basic concept
Publisher
ConsumerQueue
Queue
binding
binding
Exchange
RabbitMQ server
Basic concept
Consumer
Consumer
Consumer
Publisher
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Basic concept
statisticsqueue
webhookqueue
#
#.company_15
dealsexchange
Deals service
statistics service
webhook service
webhook service
Queue
Queue
binding
binding
Exchange
Consumer
Consumer
Consumer
RabbitMQ server
Basic concept
Publisher
Publishing
Publishing
➔ Your apps publish messages. Each message has
◆ routing key (e.g. subject)
◆ delivery mode (persistent or not persistent)
◆ headers (e.g. { "content-type": "application/json" })
◆ payload (e.g. { "some": "json" })
◆ properties (e.g. correlation-id)
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Consumer
Consumer
Consumer
Basic concept
Publisher
Publishing
RabbitMQ server
Exchange
Basic concept
Consumer
Consumer
Consumer
Publisher
Queue
Queue
binding
binding
Exchange
Exchanges
➔ Exchanges are AMQP entities where messages are sent to.
➔ Exchanges on their own are useless,unless bound to something.
https://www.rabbitmq.com/tutorials/amqp-concepts.html
Four types of exchanges
➔ Direct, Fanout, Topic, Headers
➔ Type defines how you can bind from it
➔ Most useful is Topic (allows regexp-like bindings)
➔ Read more from RabbitMQ docs
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Exchange
Basic concept
Consumer
Consumer
Consumer
Publisher
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Basic concept
Bindings
Consumer
Consumer
Consumer
Publisher
Bindings
➔ Rules with which messages are routedfrom exchanges to queues
➔ Each queue and exchange can have multiple bindings
➔ Examples: [characteristic].[color].[kind]◆ *.orange.*◆ *.*.rabbit◆ lazy.#
Bindings
Q1 is interested in all the orange animals.
Q2 wants to hear everything about rabbits, and everything about lazy animals.
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Basic concept
Bindings
Consumer
Consumer
Consumer
Publisher
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Basic concept
Consumer
Consumer
Consumer
Publisher
Queues
Queues
➔ Ordered lists of messages to be consumed
➔ FIFO
➔ Can be either durable or transient
➔ Can have arguments (behavioral properties)TTL of messages in itauto-expirymaximum lengthdead letter handling
Queue
Queue
binding
binding
Exchange
RabbitMQ server
Basic concept
Consumer
Consumer
Consumer
Publisher
Queues
RabbitMQ server
Basic concept
Queue
Queue
binding
binding
Exchange
Consuming
Consumer
Consumer
Consumer
Publisher
Consuming
➔ Act of receiving and acting upon messages from a queue
Consuming
➔ Act of receiving and acting upon messages from a queue
➔ One at a time, or multiple-in-flight with acking
Consuming
➔ Act of receiving and acting upon messages from a queue
➔ One at a time, or multiple-in-flight with acking
➔ Delivery and execution guarantees
Consuming
➔ Act of receiving and acting upon messages from a queue
➔ One at a time, or multiple-in-flight with acking
➔ Delivery and execution guarantees
➔ One queue can have multiple consumers (enables load balancing — but then exact order is not guaranteed)
Consuming
➔ Act of receiving and acting upon messages from a queue
➔ One at a time, or multiple-in-flight with acking
➔ Delivery and execution guarantees
➔ One queue can have multiple consumers (enables load balancing — but then exact order is not guaranteed)
➔ A consumer may ask for exclusivity
Consumption sequence diagram
RabbitMQ Consumer
RabbitMQ Consumer
[message]
Consumption sequence diagram
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
Consumption sequence diagram
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’ attempt to process
the message
Consumption sequence diagram
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’ attempt to process
the messageack
Consumption sequence diagram
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’ attempt to process
the messageack
mark message delivered
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
attempt to process the message
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
attempt to process the message
dies!
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
attempt to process the message
(connection reset) dies!
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
attempt to process the message
(connection reset) dies!
put message back to ‘ready’
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
attempt to process the message
(connection reset) dies!
put message back to ‘ready’
Consumer
Consumer
Consumption sequence diagram with a failure
RabbitMQ Consumer
RabbitMQ Consumer
[message]mark
message ‘unacked’
attempt to process the message
(connection reset) dies!
put message back to ‘ready’
Consumer
Consumer
[message]mark message ‘unacked’
Consumer characteristics (Node.js)
➔ Persistent TCP connection to AMQP server
➔ RabbitMQ pushes messages to consumer (no polling)
➔ Connection is multiplexed (using channels)
➔ Event listener/callback function is executed per each message, with ack() callback provided to be called when a message can be considered consumed.
RabbitMQ server
Basic concept
Publisher
Queue
Queue
binding
binding
Consumer
Consumer
Consumer
Exchange
RabbitMQ server
Basic concept
Deals service
statisticsqueue
webhookqueue
#
#.company_15
statistics service
webhook service
dealsexchange
webhook service
RabbitMQ server
Basic concept
Deals service
dealsexchange
Photos service
statisticsqueue
webhookqueue
statistics service
webhook service
webhook service
photosexchange
RabbitMQ server
Basic concept
Deals service
statisticsqueue
webhookqueue
statistics service
webhook service
dealsexchange
webhook service
Photos service
photosexchange
analyzerqueue
analyzer service
RabbitMQ server
Basic concept
Deals service
statisticsqueue
webhookqueue
statistics service
webhook service
dealsexchange
webhook service
Photos service
photosexchange
analyzerqueue
analyzer service
resizingqueue photo resizer
service
So it’s like…a series of tubes.
Some like to call it“a post box, a post office and a postman, all in one”...
“When you send mail to the post box you're pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman.”From the RabbitMQ official tutorial.
However,in a microservices environment, this post metaphor can be dangerous.
The post metaphor can lure you into publishing messages with a specific consumer in mind,
thus increasing tight coupling which makes it harder to add other kinds of consumers in the future without having to change multiple services.
Example:Orchestration vs choreography
User signs up
Example use case shown by Sam Newman in “Principles of microservices” presentation
User signs up
Create user account
Example use case shown by Sam Newman in “Principles of microservices” presentation
User signs up
Create user account
Sign up to newsletter at MailChimp
Example use case shown by Sam Newman in “Principles of microservices” presentation
User signs up
Create user account
Sign up to newsletter at MailChimp
Send welcome email
Example use case shown by Sam Newman in “Principles of microservices” presentation
User signs up
Create user account
Sign up to newsletter at MailChimp
Send welcome email Send welcome gift
Example use case shown by Sam Newman in “Principles of microservices” presentation
User signs up
Create user account
Sign up to newsletter at MailChimp
Send welcome email Send welcome gift
Example use case shown by Sam Newman in “Principles of microservices” presentation
Done!
Orchestration
Example use case shown by Sam Newman in “Principles of microservices” presentation
Sign up service
Orchestration
Example use case shown by Sam Newman in “Principles of microservices” presentation
Orchestration
Sign up service
Newsletter service
subscribe
Example use case shown by Sam Newman in “Principles of microservices” presentation
Orchestration
Sign up service
Newsletter service
Email sendersubscribe
send email
Example use case shown by Sam Newman in “Principles of microservices” presentation
Orchestration
Sign up service
Newsletter service
Email sender
Delivery service
subscribe
send email
send package
➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind.
Problem with orchestration — tight coupling
➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind.
➔ Makes it hard to plug in new services along the way, and thus shape the system organically.
Problem with orchestration — tight coupling
➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind.
➔ Makes it hard to plug in new services along the way, and thus shape the system organically.
➔ Expectations of replies to completed commands — seeing a “send this newsletter out” command in service A may seem like it is a synchronous action, whereas with a message queue it is not.
Problem with orchestration — tight coupling
Choreography
https://www.flickr.com/photos/stevenpisano/16313427202
Choreography
Sign up service
Example use case shown by Sam Newman in “Principles of microservices” presentation
Example use case shown by Sam Newman in “Principles of microservices” presentation
Choreography
Sign up service
user signed up
Example use case shown by Sam Newman in “Principles of microservices” presentation
Choreography
Sign up service
Newsletter service
user signed up
Example use case shown by Sam Newman in “Principles of microservices” presentation
Choreography
Sign up service
Email sender
Newsletter service
user signed up
Example use case shown by Sam Newman in “Principles of microservices” presentation
Choreography
Sign up service
Email sender
Delivery service
Newsletter service
user signed up
Choreography
Sign up service
Newsletter service
Email sender
Delivery service
RabbitMQ
Queue
Queue
signups Queue
Example use case shown by Sam Newman in “Principles of microservices” presentation
➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.
Choreography
➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.
➔ ResilientFailure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed.
Choreography
➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.
➔ ResilientFailure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed.
➔ ElasticQueue lengths can be used to scale consuming services up or down.
Choreography
➔ ResponsiveUser does not have to wait for all services to complete their work, thus gets the response faster.
➔ ResilientFailure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed.
➔ ElasticQueue lengths can be used to scale consuming services up or down.
➔ Works well with message queues. Allows reactive architectures.
Choreography
So, embrace choreography wherever possible.
Dumb pipes,smart endpoints
Dumb service A
Dumb service B
Dumb service C
Dumb service D
Dumb service A
Dumb service B
Dumb service C
Dumb service D
Enterprise Service Bus
Dumb service A
Dumb service B
Dumb service C
Dumb service D
Dumb service A
Dumb service B
Dumb service C
Dumb service D
Magic Mystery Bus
Example credit to Sam Newman in “Principles of microservices” presentation.
Dumb service A
Dumb service B
Dumb service C
Dumb service D
Dumb service A
Dumb service B
Dumb service C
Dumb service D
Magic Mystery Bus
Dumb pipes, smart endpoints
➔ Initially, you deploy an empty RabbitMQ.
Dumb pipes, smart endpoints
➔ Initially, you deploy an empty RabbitMQ.
➔ Do not make RabbitMQ deployment be aware of the desired data flows.
Dumb pipes, smart endpoints
➔ Initially, you deploy an empty RabbitMQ.
➔ Do not make RabbitMQ deployment be aware of the desired data flows.
➔ Exchanges, bindings and queues are created by services themselves as they need.
RabbitMQ server
Basic concept
Deals service
statisticsqueue
webhookqueue
#
#.company_15
statistics service
webhook service
dealsexchange
webhook service
RabbitMQ server
Basic concept political map
Deals service
statisticsqueue
webhookqueue
#
#.company_15
statistics service
webhook service
dealsexchange
webhook service
RabbitMQ server
Basic concept political map
Deals service
statisticsqueue
webhookqueue
#
#.company_15
statistics service
webhook service
dealsexchange
webhook serviceKingdom of
Deals Service
Commonwealth of Statistics Service
Republic of Webhook Service
So, in essence, RabbitMQ is left with nothing but the dumb pipes.
Smart service A
Smart service B
Smart service C
Smart service D
Smart service A
Smart service B
Smart service C
Smart service D
Dumb pipes
RabbitMQin the wild
RabbitMQ at Pipedrive
➔ Started using in 2012
RabbitMQ at Pipedrive
➔ Started using in 2012➔ Main backbone of async service-to-service communication
RabbitMQ at Pipedrive
➔ Started using in 2012➔ Main backbone of async service-to-service communication➔ All data change events are published to RabbitMQ.
RabbitMQ at Pipedrive
➔ Started using in 2012➔ Main backbone of async service-to-service communication➔ All data change events are published to RabbitMQ.➔ Averaging around >1,700 msg/sec. Peaking sometimes at
3,000+ msg/sec. >140M msg/day.
RabbitMQ at Pipedrive
➔ Started using in 2012➔ Main backbone of async service-to-service communication➔ All data change events are published to RabbitMQ.➔ Averaging around >1,700 msg/sec. Peaking sometimes at
3,000+ msg/sec. >140M msg/day.
RabbitMQ at Pipedrive
RabbitMQ at Pipedrive
➔ Initially deployed a 3-node cluster
MQ
MQ
MQ
RabbitMQ at Pipedrive
➔ Initially deployed a 3-node cluster➔ Then scaled it up to a 5-node cluster
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
RabbitMQ at Pipedrive
➔ Initially deployed a 3-node cluster➔ Then scaled it up to a 5-node cluster➔ Then decided to move to multiple 3-node clusters instead
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
MQ
RabbitMQ at Pipedrive
RabbitMQ at Pipedrive
➔ Started seeing networking overhead in a 5-node cluster as the load grew.
RabbitMQ at Pipedrive
➔ Started seeing networking overhead in a 5-node cluster as the load grew.
➔ Multiple clusters with semi-isolated traffic (bound by either SLA or logical service groups) was easier to handle for us.
RabbitMQ at Pipedrive
➔ Started seeing networking overhead in a 5-node cluster as the load grew.
➔ Multiple clusters with semi-isolated traffic (bound by either SLA or logical service groups) was easier to handle for us.
➔ It was a logical architecture decision, not a technical constraint — RabbitMQ does scale well beyond our load.
3 caveats
1.Who owns an exchange?
Exchange ownership question
Exchange ownership question
➔ Ultimately, every service will depend on existence of certain exchanges.
Exchange ownership question
➔ Ultimately, every service will depend on existence of certain exchanges.
➔ If these are missing, binding will throw an error.
Exchange ownership question
➔ Ultimately, every service will depend on existence of certain exchanges.
➔ If these are missing, binding will throw an error.
➔ But which service should create the exchange?There should ideally be only one owner per each exchange.
Exchange ownership question
➔ Ultimately, every service will depend on existence of certain exchanges.
➔ If these are missing, binding will throw an error.
➔ But which service should create the exchange?There should ideally be only one owner per each exchange.
➔ At Pipedrive, we have so far had shared exchange ownerships. Eventually it could create problems down the line as exchange properties are defined in multiple services.
Exchange ownership question
➔ Ultimately, every service will depend on existence of certain exchanges.
➔ If these are missing, binding will throw an error.
➔ But which service should create the exchange?There should ideally be only one owner per each exchange.
➔ At Pipedrive, we have so far had shared exchange ownerships. Eventually it could create problems down the line as exchange properties are defined in multiple services.
➔ Possible solution: die consumers when an exchange does not exist?
2.Adding/removing bindings is like changing a database schema
Adding/removing bindings is like changing schema
Adding/removing bindings is like changing schema
➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.
Adding/removing bindings is like changing schema
➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.
➔ It takes time.
Adding/removing bindings is like changing schema
➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.
➔ It takes time.
➔ At Pipedrive, we used to create a queue+binding for each logged in user to facilitate websocket connection back to end user’s browser.
Adding/removing bindings is like changing schema
➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.
➔ It takes time.
➔ At Pipedrive, we used to create a queue+binding for each logged in user to facilitate websocket connection back to end user’s browser.
➔ Don’t do it.
tl;dr:RabbitMQ is optimized for throughput of messages, not for throughput of schema changes on the fly.
At Pipedrive, we have a microservice we call socketqueue.
It’s a piping service betweenuser’s browser and RabbitMQ
websocket User’s browser
socketqueue(consumer)
RabbitMQ
REST APIs(publishers)
AMQPAMQP
Socketqueue 1.0 architecture
➔ One queue + binding per each connected user
➔ One Websocket connection per each connected user
➔ One-to-one relation between queue and websocket inside the socketqueue service.
➔ Horizontally scalable
RabbitMQ cluster
Socketqueue 1.0 architecture
REST APIs
q3
api.eventsexchange
q2
q1
socketq1
u1u2u3
q6
q5
q4
socketq2
u4u5u6
q8 socketq3
u7u8u9
q7
+ 10K more
q9
300+ msg/sec
RabbitMQ cluster
Socketqueue 1.0 architecture
REST APIs
q3
api.eventsexchange
q2
q1
socketq1
u1u2u3
q6
q5
q4
socketq2
u4u5u6
q8 socketq3
u7u8u9
q7
+ 10K more
q9
Did not scale beyond 10K queues, bindings
300+ msg/sec
RabbitMQ cluster
Socketqueue 1.0 architecture
REST APIs
q3
api.eventsexchange
q2
q1
socketq1
u1u2u3
q6
q5
q4
socketq2
u4u5u6
q8 socketq3
u7u8u9
q7
+ 10K more
q9
300+ msg/sec
Socketqueue 2.0 architecture
REST APIs api.eventsexchange
q-socket1 socketq1
u1u2u3
q-socket2 socketq2
u4u5u6
socketq3
u7u8u9
q-socket3
RabbitMQ cluster
300+ msg/sec
Socketqueue 1.0 architecture
➔ One queue + binding per each connected user
➔ One Websocket connection per each connected user
➔ One-to-one relation between queue and websocket inside the socketqueue service.
➔ Horizontally scalable
Socketqueue 1.02.0 architecture
➔ One queue + binding per each connected user socketqueue service instance.
➔ One Websocket connection per each connected user
➔ One-to-onemany relation between queue and websocket inside the socketqueue service.
➔ Horizontally scalable well beyond 10K simultaneous users
Most problematic?
Initial setup of 10K+ queues/bindings
during version upgrades, service restarts.
3.Delivery guarantees
100% delivery guarantee
=
Publisher Acknowledgements
Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ
Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk
Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message
Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue
Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue➔ A consumer picks up the message and starts processing it
Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue➔ A consumer picks up the message and starts processing it➔ RabbitMQ server dies and comes back up
Publisher Acknowledgements➔ Your service sends a persistent message to RabbitMQ➔ RabbitMQ receives your message but does not yet write it to disk➔ RabbitMQ tells you it has received your message➔ The message gets routed to a durable queue➔ A consumer picks up the message and starts processing it➔ RabbitMQ server dies and comes back up➔ Except, that message that was being processed? It’s not there any more.
100% delivery guarantee
=
Publisher Acknowledgements
Read more from https://www.rabbitmq.com/confirms.html
RabbitMQ elsewhere
https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine
1 millionmessages per second
1,000,000messages per second
86,400,000,000messages per day
To put this into context...
11B2014 all of SMS traffic
per day globally
http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
17.2B
11B2014 all of SMS traffic
per day globally
2016 Apple iMessages per day globally
http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
17.2B
11B2014 all of SMS traffic
per day globally
2016 Apple iMessages per day globally
Facebook, Messenger, and WhatsApp per day
combined
45B
http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
86.4B
17.2B
11B2014 all of SMS traffic
per day globally
2016 Apple iMessages per day globally
That single RabbitMQ experiment
Facebook, Messenger, and WhatsApp per day
combined
45B
http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statisticshttp://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
I think that’s pretty remarkable.
Thank you.
Martin Tajur@tajur on Twitter
With this promo code
DEVCLUBEE2 months freewww.pipedrive.com
Thank you.
Martin Tajur@tajur on Twitter
With this promo code
DEVCLUBEE2 months freewww.pipedrive.com
We’re hiring.
Questions/Answers