kafkaとaws kinesisの比較

Kafka AWS Kinesis

/ @laclefyoshi / ysaeki@r.recruit.co.jp

• • Kafka Kinesis

• • •

•

2

• 2011/04

• 2015/09

• • Druid (2015; Kindle Direct Publishing) • RDB NoSQL (2016; ; HBase )

• • (WebDB Forum 2014)

• Spark Streaming (Spark Meetup

December 2015)

3

Kafka Kinesis

• Apache • Publish-Subscribe messaging rethought as a

distributed commit log

• AWS Kinesis • Stream • Firehose • Analytics

: Pub/Sub : : (SQL)

4

Kafka Kinesis

• Apache • Publish-Subscribe messaging rethought as a

distributed commit log

• AWS Kinesis • Stream • Firehose • Analytics

: Pub/Sub : : (SQL)

4

:

5

: <100ms

: ~8000msgs/sec

Kinesis

[Kafka] [Kinesis]

6

[Kafka] [Kinesis]

6

[Kafka] [Kinesis]

Kafka Connect

Kafka-rest Kafka-Pixy Kastle

AWS API Gateway

HTTP API

ETL ETL 7

[Kafka] [Kinesis]

Kafka Connect

Kafka-rest Kafka-Pixy Kastle

AWS API Gateway

HTTP API

ETL ETL

OSS

•Kafka Streams •PipelineDB

AWS

•Kinesis Analytics

7

[Kafka] [Kinesis]

JMX Reporter

Kerberos

Cloudwatch

AWS

7

← →

← →

← →

8

[Kafka] [Kinesis]

JMX Reporter

Kerberos

Cloudwatch

AWS

7

← →

← →

← →

8

[ ]

http://insightdataengineering.com/blog/ingestion-comparison/

X

Kafka Kinesis

• Kafka or Kinesis

• Kafka OSS

Kinesis

•

9

Kafka Kinesis

• (< 5000msgs/sec) Kinesis

• Kafka

• Kinesis

• Hadoop Kafka

• Kinesis

• Kafka

10

• OSS → Kafka

• → Kinesis

• API

• • SNS Bot IoT

•

11