dissecting the rabbit: rabbitmq internal architecture
DESCRIPTION
Talk given at HighLoad++ in Moscow, 2013. Introduces several Erlang patterns and techniques used by RabbitMQTRANSCRIPT
![Page 1: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/1.jpg)
Dissecting the Rabbit: RabbitMQ Internal Architecture
Alvaro Videla - RabbitMQ
![Page 2: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/2.jpg)
Alvaro Videla• Developer Advocate at Pivotal / RabbitMQ!
• Co-Author of RabbitMQ in Action!
• Creator of the RabbitMQ Simulator!
• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!
• @old_sound | [email protected]/videlalvaro
![Page 4: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/4.jpg)
Agenda• Intro to RabbitMQ
• Dive into RabbitMQ Internals
• A day in the life of a message
• RabbitMQ message store
• RabbitMQ behaviours and extensibility
![Page 5: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/5.jpg)
What is RabbitMQ
![Page 6: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/6.jpg)
RabbitMQ
![Page 7: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/7.jpg)
RabbitMQ
![Page 8: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/8.jpg)
RabbitMQ• Multi Protocol Messaging Server
![Page 9: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/9.jpg)
• Multi Protocol Messaging Server!• Open Source (MPL)
RabbitMQ
![Page 10: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/10.jpg)
• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot
RabbitMQ
![Page 11: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/11.jpg)
• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot!• Written in Erlang/OTP
RabbitMQ
![Page 12: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/12.jpg)
Multi Protocol
http://bit.ly/rmq-protocols
![Page 13: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/13.jpg)
Polyglot
![Page 14: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/14.jpg)
Polyglot
![Page 15: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/15.jpg)
Polyglot
• Java
![Page 16: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/16.jpg)
Polyglot
• Java!• node.js
![Page 17: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/17.jpg)
Polyglot
• Java!• node.js!• Erlang
![Page 18: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/18.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP
![Page 19: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/19.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby
![Page 20: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/20.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net
![Page 21: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/21.jpg)
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net!• Haskell
![Page 22: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/22.jpg)
Polyglot
Even COBOL!!!11
![Page 23: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/23.jpg)
Some users of RabbitMQ
![Page 24: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/24.jpg)
Some users of RabbitMQ
![Page 25: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/25.jpg)
• Instagram!• Indeed.com
Some users of RabbitMQ
![Page 26: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/26.jpg)
• Instagram!• Indeed.com!• MailboxApp
Some users of RabbitMQ
![Page 27: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/27.jpg)
• Instagram!• Indeed.com!• MailboxApp!• Mercado Libre
Some users of RabbitMQ
![Page 28: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/28.jpg)
• Instagram!• Indeed.com!• MailboxApp!• Mercado Libre!• NHS
Some users of RabbitMQ
![Page 29: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/29.jpg)
• Instagram!• Indeed.com!• MailboxApp!• Mercado Libre!• NHS!• Mozilla
Some users of RabbitMQ
![Page 30: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/30.jpg)
http://www.rabbitmq.com/download.html
Unix - Mac - Windows
![Page 31: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/31.jpg)
Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator
![Page 33: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/33.jpg)
RabbitMQ Simulator
![Page 34: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/34.jpg)
RabbitMQ Internals
![Page 35: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/35.jpg)
A day in the life of a message
![Page 36: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/36.jpg)
A day in the life of a message
![Page 37: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/37.jpg)
A day in the life of a message$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);!$ch = $conn->channel();!!
$ch->queue_declare($queue, false, true, false, false);!!
$ch->exchange_declare($exchange, 'direct', false, true, false);!!
$ch->queue_bind($queue, $exchange);!!
$msg_body = implode(' ', array_slice($argv, 1));!$msg = new AMQPMessage($msg_body, array('delivery_mode' => 2));!!
$ch->basic_publish($msg, $exchange);
![Page 38: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/38.jpg)
What happens here?
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);!$ch = $conn->channel();
![Page 39: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/39.jpg)
What happens here?
![Page 40: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/40.jpg)
Erlang
![Page 41: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/41.jpg)
Erlang App
• Processes (probably thousands)
• They communicate sending messages
• Each process has a message queue (don’t confuse with RabbitMQ queues)
• Virtual Machine has preemptive scheduler
![Page 42: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/42.jpg)
Read More Here:
http://jlouisramblings.blogspot.ru/2013/01/how-erlang-does-scheduling.html
![Page 43: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/43.jpg)
Erlang code structure• Modules
• Functions
• Function Arity
• Arguments
M, F, A = Module, Function, Arguments
![Page 44: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/44.jpg)
rabbit_client_sup.erl-module(rabbit_client_sup).!!
-behaviour(supervisor2).!!
-export([start_link/1, start_link/2, start_link_worker/2]).!!
-export([init/1]).!!
-include("rabbit.hrl").
![Page 45: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/45.jpg)
rabbit_client_sup.erlstart_link(Callback) ->! supervisor2:start_link(?MODULE, Callback).!!
start_link(SupName, Callback) ->! supervisor2:start_link(SupName, ?MODULE, Callback).!!
start_link_worker(SupName, Callback) ->! supervisor2:start_link(SupName, ?MODULE, {Callback, worker}).!!
init({M,F,A}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, infinity, supervisor, [M]}]}};!init({{M,F,A}, worker}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, ?MAX_WAIT, worker, [M]}]}}.
![Page 46: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/46.jpg)
Supervision Trees
![Page 47: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/47.jpg)
Supervision tree
• Worker Processes
• Supervisor Processes
• Supervision tree as a hierarchical arrangement of processes
http://www.erlang.org/doc/man/supervisor.html
![Page 48: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/48.jpg)
rabbit_client_sup
init({M,F,A}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, !! ! ! ! infinity, supervisor, [M]}]}};
![Page 49: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/49.jpg)
Child Spec - restart strategies• one_for_one: only restart failing process
• one_for_all: restart failing process and all siblings
• simple_one_for_one: simplified version of one_for_one
• MaxR: maximum restarts allowed
• MaxT: in MaxT seconds
![Page 50: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/50.jpg)
Child Specificationchild_spec() = {Id,StartFunc,Restart,Shutdown,Type,Modules}! Id = term()! StartFunc = {M,F,A}! M = F = atom()! A = [term()]! Restart = permanent | transient | temporary! Shutdown = brutal_kill | int()>0 | infinity! Type = worker | supervisor! Modules = [Module] | dynamic! Module = atom()
![Page 51: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/51.jpg)
rabbit_client_sup
init({M,F,A}) ->! {ok, {{simple_one_for_one, 0, 1},! [{client, {M,F,A}, temporary, !! ! ! ! infinity, supervisor, [M]}]}};
![Page 52: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/52.jpg)
Child Spec - restart
• permanent: should always be restarted
• temporary: should never be restarted
• transient: should only be restarted if terminated abnormally
![Page 53: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/53.jpg)
Child Spec - shutdown• brutal_kill: child is terminated immediately.
• timeout in seconds: supervisor waits for timeout before terminating children.
• infinity: give enough timeout to children to shutdown its own supervision tree.
![Page 54: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/54.jpg)
Connection Supervision Tree
![Page 55: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/55.jpg)
Connection Supervision Tree
![Page 56: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/56.jpg)
Connection Supervision Tree
![Page 57: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/57.jpg)
Connection Supervision Tree
![Page 58: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/58.jpg)
Connection Supervision Tree
![Page 59: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/59.jpg)
A day in the life of a message
![Page 60: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/60.jpg)
A day in the life of a message
![Page 61: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/61.jpg)
A day in the life of a message
![Page 62: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/62.jpg)
Intermezzo: pattern matchingcheck_user_id_header(#'P_basic'{user_id = undefined}, _) ->! ok;!check_user_id_header(#'P_basic'{user_id = Username},! #ch{user = #user{username = Username}}) ->! ok;!check_user_id_header(#'P_basic'{user_id = Claimed},! #ch{user = #user{username = Actual,! tags = Tags}}) ->! case lists:member(impersonator, Tags) of! true -> ok;! false -> precondition_failed(! "user_id property set to '~s' but authenticated user was "! "'~s'", [Claimed, Actual])! end.
http://videlalvaro.github.io/2013/09/rabbitmq-validating-user-ids-with-erlang-pattern-matching.html
![Page 63: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/63.jpg)
Read More Here:
http://videlalvaro.github.io/2013/09/rabbitmq-validating-user-ids-with-erlang-pattern-matching.html
![Page 64: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/64.jpg)
A day in the life of a message
![Page 65: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/65.jpg)
A day in the life of a message
![Page 66: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/66.jpg)
A day in the life of a message
We have a list of queues where the channel will deliver the messages
![Page 67: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/67.jpg)
A day in the life of a message
![Page 68: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/68.jpg)
A day in the life of a message
If consumer ready
![Page 69: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/69.jpg)
RabbitMQ Message Store• Made specifically for messaging
• Keeps messages in memory and (sometimes) on disk
• Bounded by disk size
• per node message store (transient, persistent)
• per queue “queue index”
![Page 70: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/70.jpg)
RabbitMQ Message Store
• Message written to disk when:
• Message published as persistent (delivery_mode = 2)
• Memory pressure
![Page 71: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/71.jpg)
Read More Here:
![Page 72: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/72.jpg)
Read More Here:Why a custom message store?
!
http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/
![Page 73: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/73.jpg)
Read More Here:Why a custom message store?
!
http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/
How the message store compacts files !
http://hg.rabbitmq.com/rabbitmq-server/file/56d190fd4ea3/src/rabbit_msg_store.erl#l181
![Page 74: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/74.jpg)
Read More Here:Why a custom message store?
!
http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/
How the message store compacts files !
http://hg.rabbitmq.com/rabbitmq-server/file/56d190fd4ea3/src/rabbit_msg_store.erl#l181
How the message store reacts to memory pressure !
http://hg.rabbitmq.com/rabbitmq-server/file/56d190fd4ea3/src/rabbit_variable_queue.erl#l35
![Page 75: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/75.jpg)
Credit Flow
![Page 76: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/76.jpg)
Prevent processes from overflowing each other’s mailboxes
![Page 77: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/77.jpg)
Credit Specification
{InitialCredit, MoreCreditAfter}
![Page 78: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/78.jpg)
Credit Specification
{200, 50}
![Page 79: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/79.jpg)
Process A sends messages to B
![Page 80: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/80.jpg)
Process will
• Grant more credit
• Block other processes
• Delay granting credits
![Page 81: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/81.jpg)
Read More Here:
http://videlalvaro.github.io/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html
![Page 82: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/82.jpg)
RabbitMQ Behaviours
![Page 83: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/83.jpg)
RabbitMQ Behaviours• Add a common interface for different components
![Page 84: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/84.jpg)
RabbitMQ Behaviours• Add a common interface for different components
• Exchanges
![Page 85: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/85.jpg)
RabbitMQ Behaviours• Add a common interface for different components
• Exchanges
• Queues
![Page 86: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/86.jpg)
RabbitMQ Behaviours• Add a common interface for different components
• Exchanges
• Queues
• Decorators
![Page 87: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/87.jpg)
RabbitMQ Behaviours• Add a common interface for different components
• Exchanges
• Queues
• Decorators
• Authentication methods
![Page 88: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/88.jpg)
RabbitMQ Behaviours• Add a common interface for different components
• Exchanges
• Queues
• Decorators
• Authentication methods
• Add extensibility
![Page 89: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/89.jpg)
rabbit_exchange_type-module(rabbit_exchange_type).!!-callback description() -> [proplists:property()].!!-callback serialise_events() -> boolean().!!-callback route(rabbit_types:exchange(), rabbit_types:delivery()) ->! rabbit_router:match_result().!!-callback validate(rabbit_types:exchange()) -> 'ok'.!!-callback validate_binding(rabbit_types:exchange(), rabbit_types:binding()) ->! rabbit_types:ok_or_error({'binding_invalid', string(), [any()]}).
![Page 90: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/90.jpg)
• You can add your own exchange type via plugins
• consistent hash exchange
• random exchange
• recent history exchange
• riak exchange
rabbit_exchange_type
![Page 91: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/91.jpg)
RabbitMQ Behaviours• rabbit_auth_backend (config) • rabbit_msg_store_index (config) • rabbit_backing_queue (config)
![Page 92: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/92.jpg)
RabbitMQ Behaviours• rabbit_auth_backend (config) • rabbit_msg_store_index (config) • rabbit_backing_queue (config)
[{rabbit,! [{auth_backends, [rabbit_auth_backend_http, ! rabbit_auth_backend_internal]}]! }].
![Page 93: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/93.jpg)
RabbitMQ Behaviours• rabbit_auth_mechanism (registry) • rabbit_exchange_decorator (registry) • rabbit_exchange_type (registry) • rabbit_mirror_queue_mode (registry) • rabbit_policy_validator (registry) • rabbit_queue_decorator (registry) • rabbit_runtime_parameter (registry)
![Page 94: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/94.jpg)
RabbitMQ Behaviours• rabbit_auth_mechanism (registry) • rabbit_exchange_decorator (registry) • rabbit_exchange_type (registry) • rabbit_mirror_queue_mode (registry) • rabbit_policy_validator (registry) • rabbit_queue_decorator (registry) • rabbit_runtime_parameter (registry)
-rabbit_boot_step({?MODULE,! [{description, "exchange type direct"},! {mfa, {rabbit_registry, register,! [exchange, <<"direct">>, ?MODULE]}},! {requires, rabbit_registry},! {enables, kernel_ready}]}).
![Page 95: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/95.jpg)
RabbitMQ Federation Plugin• Message replication across WANs
• Uses an exchange decorator
• Uses a queue decorator
• Uses parameters
• Uses policies
![Page 96: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/96.jpg)
RabbitMQ Federation Plugin
rabbitmqctl set_parameter federation-upstream my-upstream \!'{"uri":"amqp://server-name","expires":3600000}'
![Page 97: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/97.jpg)
RabbitMQ Federation Plugin
rabbitmqctl set_parameter federation-upstream my-upstream \!'{"uri":"amqp://server-name","expires":3600000}'
rabbitmqctl set_policy federate-me "^amq\." '{"federation-upstream-set":"all"}'
![Page 98: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/98.jpg)
RabbitMQ Federation Plugin
-module(rabbit_federation_queue).!!
-rabbit_boot_step({?MODULE,! [{description, "federation queue decorator"},! {mfa, {rabbit_registry, register,! [queue_decorator, <<"federation">>, ?MODULE]}},! {requires, rabbit_registry},! {enables, recovery}]}).!!
-behaviour(rabbit_queue_decorator).
![Page 99: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/99.jpg)
RabbitMQ Federation Plugin
-module(rabbit_federation_exchange).!!
-rabbit_boot_step({?MODULE,! [{description, "federation exchange decorator"},! {mfa, {rabbit_registry, register,! [exchange_decorator, <<"federation">>, ?MODULE]}},! {requires, rabbit_registry},! {enables, recovery}]}).!!
-behaviour(rabbit_exchange_decorator).
![Page 100: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/100.jpg)
Read More Here:
http://www.rabbitmq.com/federation.html
![Page 101: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/101.jpg)
Read More Here:Making sure the user provided the right behaviour: !
http://videlalvaro.github.io/2013/09/rabbitmq-internals-validating-erlang-behaviours.html
How RabbitMQ prevents arbitrary code execution: !
http://videlalvaro.github.io/2013/09/rabbitmq-sanitzing-user-input-in-erlang.html
RabbitMQ Boot Step System: !
https://github.com/videlalvaro/rabbit-internals/blob/master/rabbit_boot_process.md
![Page 102: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/102.jpg)
RabbitMQ uses Erlang’s features to be robust, fault tolerant and very
extensible
![Page 103: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/103.jpg)
Go grab the source code!
http://hg.rabbitmq.com/rabbitmq-server/
![Page 104: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/104.jpg)
Questions?
![Page 105: Dissecting the rabbit: RabbitMQ Internal Architecture](https://reader037.vdocuments.net/reader037/viewer/2022100215/554f5f33b4c9058a148b4653/html5/thumbnails/105.jpg)
ThanksAlvaro Videla - @old_sound