zeromq zookeeper and flatbuffers
TRANSCRIPT
ZeroMQ, ZooKeeper & FlatBuffers
Ravi Okade @ Code Camp NYC
Agenda:ZeroMQ Pub-SubZooKeeper Leader ElectionFlatBuffers SerializationDownload sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing
Demo Weather Application- Publishes weather info via ZeroMQ
- Publisher is a Console app. WPF and Console Subscribers.
- Starts with a simple string serialization
- Then switch to FlatBuffers for serialization
- Then add ZooKeeper:- Publishers use ZK to ensure only one publisher at a time
- Subscribers listen to ZK for any change in the publisher and re-initialize ZeroMQ
- All code runs on a single machine for the demo
- The demo is to demonstrate concepts and not to measure performance
- Code is available on Git here: TODO
ZeroMQ
http://www.slideshare.net/pieterh/overview-of-zeromqhttps://www.infoq.com/author/Pieter-Hintjenshttp://zguide.zeromq.org/page:all#Why-We-Needed-ZeroMQ
I am confused with all these products ending with MQ!- ZeroMQ, ActiveMQ, RabbitMQ, QPID, Kafka..
Some good references:
- https://www.predic8.com/activemq-hornetq-rabbitmq-apollo-qpid-comparison.htm
- http://bipinkunjumon.blogspot.com/2013/05/zeromq-and-activemq-comparison.html
- https://tomasz.janczuk.org/2015/09/from-kafka-to-zeromq-for-log-aggregation.html
ZeroMQ
http://www.slideshare.net/pieterh/overview-of-zeromq
Publisher (publishes weather data by zipcode)
Subscriber (subscribes to weather data by zipcode)
Features- Handles slow subscriber
- Resilient to publisher aborts
- Multiple language support (.NET, Java, Python..)
- Subscription filters
- Fast!
NetMQ https://github.com/zeromq/netmq- .NET port of zeromq
FlatBuffers Intro https://google.github.io/flatbuffers/https://google.github.io/flatbuffers/md__internals.html
FlatBuffers- Create Schema:
namespace codecamp;table Weather{
zipCode:string;temperature:int;humidity:int;
}
root_type Weather;
- Compile: flatc -n weather.schema
Compare FlatBuffers with:
http://real-logic.github.io/simple-binary-encodinghttps://capnproto.org/
ZooKeeperZooKeeper is a distributed, open-source coordination service for distributed applications.
ZooKeeper is very popular for leader election implementation and is widely used in the Hadoop eco-system.
What else is ZooKeeper used for?- Service discovery
- Leader election (this demo)
- Configuration repository, group membership
- Barriers
- Queue, Priority Queue
- Locks
- Two phase commit
ZooKeeper Leader Election
Publishers
Leader
ZooKeeper Service
Subscribers
Co-ordinating who is the leader
Notifying when leader changes
ZooKeeper leader election strategy- Like a DMV token system
- Zookeeper assigns an id to each publisher
- The process with lowest id is the leader
- If any process dies, Zookeeper sends a notification; the process with next lowest id becomes the leader
- Note - ZooKeeper is an ordered system and guarantees consistency; hence each id is assigned only once
ZooKeeper is also used for:- Service Discovery
- Distributed Key-value store (database)
- Configuration store
- Many more: Barriers/Locks, Queues, 2phase Commit: https://zookeeper.apache.org/doc/trunk/recipes.html
Compare ZooKeeper with:https://www.consul.io/
https://coreos.com/etcd/
Apache Curator - a framework over Zookeeper http://curator.apache.org/
Eureka (by Netflix)https://github.com/Netflix/eureka/wiki/
Referenceshttps://github.com/ewhauser/zookeeperhttps://github.com/zeromq/clrzmq (.NET Client for 0MQ)https://github.com/zeromq/clrzmq4 (newer .NET Client for 0MQ)Download sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing