/ @laclefyoshi / ysaeki@r.recruit.co.jp

• • • • • •

2

• 2011/04

• 2015/09

•

• Druid (KDP, 2015)

• RDB NoSQL ( , 2016; : HBase )

• ESP8266 Wi-Fi IoT (KDP, 2016)

• • (DEIM 2014)

• (WebDB Forum 2014)

• Spark Streaming (Spark Meetup December 2015)

• Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016)

• Kinesis Family (R-JAWS Bigdata; 2016)

3

5

• Queue Queueing system

•

• Message

• 1

• Broker

• • Producer

•

• Consumer

• • Protocol

•

7

•

• •

8

9

Apache ActiveMQ

Apache Kafka

RabbitMQ

NATS

NSQ

Redis

ZeroMQ

Nanomsg

Apache Kafka

A distributed streaming platform

10

Apache ActiveMQ

The most popular and powerful open source messaging and Integration Patterns server

11

RabbitMQ

Robust messaging for applications Easy to use

12

NATS

For cloud native applications, IoT messaging, and microservices architectures

13

NSQ

A realtime distributed messaging platform

14

Redis

In-memory data structure store, used as database, cache and message broker

15

ZeroMQ

An embeddable networking library but acts like a concurrency framework

16

Nanomsg

A socket library that provides several common communication patterns

17

19

20

: • • • Lambda Architecture

: • •

21

22

: • •

: •

23

24

: •

•

: •

25

27

: ……

https://cloud.google.com/blog/big-data/2016/09/ apache-kafka-for-gcp-users-connectors-for-pubsub-dataflow-and-bigquery

28

Pub/SubDataflow/Apache BeamBigQuery

Apache Kafka

29

30

: •

: •

31

32

: •

: •

2016/10/25

official / 3rd party33

• JIRA Github

• Apache Flink Apache Spark

Apache Bahir (3rd party)

• Apache Kafka

34

1000

• [ 1] → [ 1 → 1]

• AWS: EC2 /t2.medium x 2 ( VPC)

• github:laclefyoshi/queues-benchmark

• Docker

• Python

36

1000

37

• • •

• 1000 Byte 100000 Byte

1000 Byte

• 1000

38

40

41

Queue makes everyone happy!!