zeromq zookeeper and flatbuffers

ZeroMQ, ZooKeeper & FlatBuffers

Ravi Okade @ Code Camp NYC

Agenda:ZeroMQ Pub-SubZooKeeper Leader ElectionFlatBuffers SerializationDownload sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing

Demo Weather Application- Publishes weather info via ZeroMQ

- Publisher is a Console app. WPF and Console Subscribers.

- Starts with a simple string serialization

- Then switch to FlatBuffers for serialization

- Then add ZooKeeper:- Publishers use ZK to ensure only one publisher at a time

- Subscribers listen to ZK for any change in the publisher and re-initialize ZeroMQ

- All code runs on a single machine for the demo

- The demo is to demonstrate concepts and not to measure performance

- Code is available on Git here: TODO

ZeroMQ

http://www.slideshare.net/pieterh/overview-of-zeromqhttps://www.infoq.com/author/Pieter-Hintjenshttp://zguide.zeromq.org/page:all#Why-We-Needed-ZeroMQ

I am confused with all these products ending with MQ!- ZeroMQ, ActiveMQ, RabbitMQ, QPID, Kafka..

Some good references:

- https://www.predic8.com/activemq-hornetq-rabbitmq-apollo-qpid-comparison.htm

- http://bipinkunjumon.blogspot.com/2013/05/zeromq-and-activemq-comparison.html

- https://tomasz.janczuk.org/2015/09/from-kafka-to-zeromq-for-log-aggregation.html

ZeroMQ

http://www.slideshare.net/pieterh/overview-of-zeromq

Publisher (publishes weather data by zipcode)

Subscriber (subscribes to weather data by zipcode)

Features- Handles slow subscriber

- Resilient to publisher aborts

- Multiple language support (.NET, Java, Python..)

- Subscription filters

- Fast!

NetMQ https://github.com/zeromq/netmq- .NET port of zeromq

FlatBuffers Intro https://google.github.io/flatbuffers/https://google.github.io/flatbuffers/md__internals.html

FlatBuffers- Create Schema:

namespace codecamp;table Weather{

zipCode:string;temperature:int;humidity:int;

}

root_type Weather;

- Compile: flatc -n weather.schema

Compare FlatBuffers with:

http://real-logic.github.io/simple-binary-encodinghttps://capnproto.org/

ZooKeeperZooKeeper is a distributed, open-source coordination service for distributed applications.

ZooKeeper is very popular for leader election implementation and is widely used in the Hadoop eco-system.

What else is ZooKeeper used for?- Service discovery

- Leader election (this demo)

- Configuration repository, group membership

- Barriers

- Queue, Priority Queue

- Locks

- Two phase commit

ZooKeeper Leader Election

Publishers

Leader

ZooKeeper Service

Subscribers

Co-ordinating who is the leader

Notifying when leader changes

ZooKeeper leader election strategy- Like a DMV token system

- Zookeeper assigns an id to each publisher

- The process with lowest id is the leader

- If any process dies, Zookeeper sends a notification; the process with next lowest id becomes the leader

- Note - ZooKeeper is an ordered system and guarantees consistency; hence each id is assigned only once

ZooKeeper is also used for:- Service Discovery

- Distributed Key-value store (database)

- Configuration store

- Many more: Barriers/Locks, Queues, 2phase Commit: https://zookeeper.apache.org/doc/trunk/recipes.html

Compare ZooKeeper with:https://www.consul.io/

https://coreos.com/etcd/

Apache Curator - a framework over Zookeeper http://curator.apache.org/

Eureka (by Netflix)https://github.com/Netflix/eureka/wiki/

Referenceshttps://github.com/ewhauser/zookeeperhttps://github.com/zeromq/clrzmq (.NET Client for 0MQ)https://github.com/zeromq/clrzmq4 (newer .NET Client for 0MQ)Download sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing