scaling rabbitmq to 11
DESCRIPTION
Given at Surge 2011 in Baltimore.TRANSCRIPT
![Page 1: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/1.jpg)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
![Page 2: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/2.jpg)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
INTERACTIVE POLLS• http://pollev.com/gmr
• SMS: 22333
2
![Page 3: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/3.jpg)
Gavin M. Roy @crad - Surge 2011
myYearbook.com• #1 Comscore Teen Destination, Top Trafficked Sites in US
• Use of RabbitMQ dates back to 2009
• Multiple clusters for different purposes
• Browser <-> Backend Message Bus
• Messaging, Event Processing
3
![Page 4: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/4.jpg)
Gavin M. Roy @crad - Surge 2011
DISASTER PORN
![Page 5: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/5.jpg)
Gavin M. Roy @crad - Surge 2011
ABOUT AMPQ• Platform, Vendor Neutral Messaging format
• 0.9.1 finalized in November 2008
• JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix,
IONA Technologies, Cisco Systems, Envoy Technologies,
Twist Process Innovations
• Members now include Bank of America, Barclays Banks,
Microsoft Corporation, Novell, VMWare
• 1.0 Final: October 2011
5
![Page 6: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/6.jpg)
Gavin M. Roy @crad - Surge 2011
ABOUT RABBITMQ• Born in 2006, Released in 2007, Acquired in 2010
• Written in Erlang/OTP
• Talks AMQP 0.8 and 0.9.1
• Features include:
• Clustering, Active/Active HA Queues
• Management UI and API
• Plugin Architecture
6
![Page 7: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/7.jpg)
Gavin M. Roy @crad - Surge 2011
ABOUT YOU & RABBITMQ
![Page 8: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/8.jpg)
Gavin M. Roy @crad - Surge 2011
RABBITMQ& AMQPCONCEPTS
![Page 9: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/9.jpg)
Gavin M. Roy @crad - Surge 2011
PUBLISHERS
![Page 10: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/10.jpg)
Gavin M. Roy @crad - Surge 2011
CONSUMERS
![Page 11: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/11.jpg)
Gavin M. Roy @crad - Surge 2011
QUEUES• Durability: Queue survives reboot
• Auto-Delete: Delete the queue when consumer goes away
• Exclusive: Only one consumer may consume
• x-expires: Auto-Delete after given duration
• x-message-ttl: Auto-discard a message after ttl
11
![Page 12: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/12.jpg)
Gavin M. Roy @crad - Surge 2011
MESSAGES• Have client specified properties:
• Content type, Encoding, Timestamp, App-Id, User-Id,
Headers
• Delivery Mode:
• 1: Non-Persistent
• 2: Persistent
12
![Page 13: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/13.jpg)
Gavin M. Roy @crad - Surge 2011
MESSAGE ROUTINGIt’s not just about Queues.
![Page 14: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/14.jpg)
Gavin M. Roy @crad - Surge 2011
ROUTING KEYIn addition to the exchange, determines what is done with a
message.
![Page 15: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/15.jpg)
Gavin M. Roy @crad - Surge 2011
CC & BCCMore routing options per message, but violates AMQP.
![Page 16: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/16.jpg)
Gavin M. Roy @crad - Surge 2011
HEADERS EXCHANGEMore on this later.
![Page 17: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/17.jpg)
Gavin M. Roy @crad - Surge 2011
EXCHANGES
![Page 18: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/18.jpg)
Gavin M. Roy @crad - Surge 2011
FANOUT• 1:N Message Delivery Pattern
• No Routing Keys Involved
• Queues bound to a fanout exchange get all messages sent to
the exchange
18
![Page 19: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/19.jpg)
Gavin M. Roy @crad - Surge 2011
DIRECT• 1:1 Message Delivery Pattern
• Routing Keys are direct, no wildcarding
• All queues bound to a routing key get the message
19
![Page 20: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/20.jpg)
Gavin M. Roy @crad - Surge 2011
TOPIC EXCHANGES• Pattern Matching in Routing Keys
• Publish one message type to Surge.Sessions, another to
Surge.Mentions
• Period delimiter, * stays within the period scope, # does not
• Example patterns: *.Mentions, Surge.*, #
• Slower than Fanout and Direct
20
![Page 21: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/21.jpg)
Gavin M. Roy @crad - Surge 2011
HEADERS EXCHANGES• Matches on Basic.Properties headers table values to the
arguments table specified when binding a queue
• Use x-match when declaring exchange to specify any or all
headers must match
• Slow implementation in Erlang
21
![Page 22: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/22.jpg)
Gavin M. Roy @crad - Surge 2011
PLUGIN EXCHANGES• Consistent Hash Exchange: Round-robin distribution to queues
• External: RPC Exchange Plugin
• Script: Similar in intent to External
• Riak
• Last-Value Cache
• Recent History Exchange: Keeps last 20 messages routed
• Global Fanout: Sends to all queues, regardless of routing key
22
![Page 23: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/23.jpg)
Gavin M. Roy @crad - Surge 2011
ONE MORE THING...Exchange-to-Exchange Bindings
![Page 24: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/24.jpg)
Gavin M. Roy @crad - Surge 2011
CLUSTERS
![Page 25: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/25.jpg)
Gavin M. Roy @crad - Surge 2011
STATS, DISK AND RAM NODES
![Page 26: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/26.jpg)
Gavin M. Roy @crad - Surge 2011
WHERE DOES THE QUEUE LIVE?
![Page 27: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/27.jpg)
Gavin M. Roy @crad - Surge 2011
CROSS-NODE PUBLISHING & CONSUMING
![Page 28: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/28.jpg)
Gavin M. Roy @crad - Surge 2011
MAX CLUSTER SIZE~32 notes is a realistic maximum node count for a cluster
![Page 29: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/29.jpg)
Gavin M. Roy @crad - Surge 2011
SCALING ISSUES?
![Page 30: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/30.jpg)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
![Page 31: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/31.jpg)
Gavin M. Roy @crad - Surge 2011
WHAT TO WATCH
![Page 32: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/32.jpg)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
RABBITMQ
Y U NO TAKE MESSAGE?
![Page 33: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/33.jpg)
Gavin M. Roy @crad - Surge 2011
TCP BACKPRESSUREMUCH?
![Page 34: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/34.jpg)
Gavin M. Roy @crad - Surge 2011
MEMORY MANAGEMENT
![Page 35: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/35.jpg)
Gavin M. Roy @crad - Surge 2011
[{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}]}].
40%
![Page 36: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/36.jpg)
Gavin M. Roy @crad - Surge 2011
MESSAGE PERSISTENCE50-70% performance penalty compared to non-persisted
messages. * http://www.rabbitmq.com/faq.html#performance-persistent
![Page 37: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/37.jpg)
Gavin M. Roy @crad - Surge 2011
CONSUMINGAck, Nack and Reject vs Basic.Consume(no_ack=True)
![Page 38: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/38.jpg)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
Server Client
Basic.DeliverBasic.[Ack/Nack/Reject]
Basic.[Ack/Nack/Reject]Ok
![Page 39: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/39.jpg)
Gavin M. Roy @crad - Surge 2011
![Page 40: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/40.jpg)
Gavin M. Roy @crad - Surge 2011
BASIC.QOS
• Prefetch Size and Count
• How many messages the
client wants the RabbitMQ
to pre-deliver to the client
• Doesn’t work in no_ack
![Page 41: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/41.jpg)
Gavin M. Roy @crad - Surge 2011
TOO MANY SHORT LIVED CONNECTIONS
![Page 42: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/42.jpg)
Gavin M. Roy @crad - Surge 2011
APACHE + MOD_PHP
![Page 43: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/43.jpg)
Gavin M. Roy @crad - Surge 2011
CONNECTION PERSISTENCE
![Page 44: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/44.jpg)
Gavin M. Roy @crad - Surge 2011
100 SERVERS125 BACKENDS1 REQUEST/S12,500 CONN/S
![Page 45: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/45.jpg)
Gavin M. Roy @crad - Surge 2011
AMQP CONNECTION SEQUENCE
Client Server
Protocol Header FrameConnection.Start
Connection.StartOk
Connection.TuneOkConnection.Tune
Connection.OpenOkConnection.Open
Channel.OpenChannel.OpenOk
![Page 46: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/46.jpg)
Gavin M. Roy @crad - Surge 2011
685 BYTES PER8.16 MB/S IN STARTUP @12,500 CONN/S
![Page 47: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/47.jpg)
Gavin M. Roy @crad - Surge 2011
VORPAL BUNNY
![Page 48: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/48.jpg)
Gavin M. Roy @crad - Surge 2011
VORPAL BUNNY• Light-weight PHP client for RabbitMQ
• Used the experimental JSON-RPC Channel plugin
• Handles the overhead of the HTTP, JSON-RPC to AMQP
analog
• Fire and forget message publishing
• Can not be transparently load balanced
48
![Page 49: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/49.jpg)
Gavin M. Roy @crad - Surge 2011
TOO MANY CONNECTIONS
![Page 50: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/50.jpg)
Gavin M. Roy @crad - Surge 2011
VELOCITY TOO HIGH FOR ONE NODE?
![Page 51: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/51.jpg)
Gavin M. Roy @crad - Surge 2011
MEMORYUTILIZATION DISPROPORTIONATE TO QUEUE DEPTHS
![Page 52: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/52.jpg)
Gavin M. Roy @crad - Surge 2011
CLUSTERNODE FAILURES
![Page 53: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/53.jpg)
Gavin M. Roy @crad - Surge 2011
UP TO 60 SECONDS TO DETECT
![Page 54: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/54.jpg)
Gavin M. Roy @crad - Surge 2011
MESSAGES DROPPED ON THE FLOOR*
![Page 55: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/55.jpg)
Gavin M. Roy @crad - Surge 2011
* 2.6.0 ADDEDACTIVE/ACTIVE QUEUES IN CLUSTERS
![Page 56: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/56.jpg)
Gavin M. Roy @crad - Surge 2011
ACTIVE/PASSIVE FAILOVER?Recommendation is to use Pacemaker and DRBD or shared
storage for Mnesia databases.
![Page 57: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/57.jpg)
Gavin M. Roy @crad - Surge 2011
RECOVERY?
![Page 58: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/58.jpg)
Gavin M. Roy @crad - Surge 2011
CLUSTERING ALTERNATIVES
![Page 59: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/59.jpg)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQ to 11
FEDERATION
• Exchange Plugin
• Provides ability to route all
messages to a x-federation
exchange to another broker
![Page 60: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/60.jpg)
Gavin M. Roy @crad - Surge 2011
SHOVEL PLUGIN
• From Queue on one broker to an
exchange on another broker.
• Effective for cross-cluster data
broadcasting
![Page 61: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/61.jpg)
Gavin M. Roy @crad - Surge 2011
TRENDING & MONITORING• RabbitMQ Message Velocity Rates Gotcha
• Management API Plugin exposes RESTful API including
monitoring data points
• 2.6.1 allows monitoring tag for users to allow data access
without full admin access
• Queue.Declare(passive=True) gets queue depths as well
• Plug in to Graphite, Nagios,or your favorite tool
61
![Page 62: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/62.jpg)
Gavin M. Roy @crad - Surge 2011
PHOTO CREDITS• “skateboard kids 7”
© 2007 Volker Neumann
http://flickr.com/prawnpie
• “methodcn contest”
©2009 Erin O’Hara
http://flickr.com/erinohara73/
• USPS Truck Accident
Valley City Fire Department
http://www.valleycityfire.com
62
![Page 63: Scaling RabbitMQ to 11](https://reader034.vdocuments.net/reader034/viewer/2022052310/553d3ade5503466f328b4588/html5/thumbnails/63.jpg)
Gavin M. Roy @crad - Surge 2011
Scaling RabbitMQto 11
QUESTIONS?