APACHE IGNITE AS A DATA PROCESSING HUB ROMAN SHTYKH

CYBERAGENT, INC.

INTRODUCTION

ABOUT ME

Roman Shtykh

¡  R&D Engineer at CyberAgent, Inc.

¡  Areas of focus

¡  Data streaming and NLP

¡  Committer on the Apache Ignite and MyBatis projects

¡  Judoka

¡  @rshtykh

CYBERAGENT, INC.

¡  Internet ads

¡  Games

¡  Media

¡  Investing

25%

13%

52%

3% 7%

Games

Media

Internet ads

Investing

Other

* As of Sep 2015

AMEBA SERVICES

・ Monthly visitors (DUB total):

6 billion* ・ Number of member users :

about 39 million*

CyberAgent, Inc.

Ameba Services

* As of Dec 2014

•  Games •  Community services •  Content curation •  Other

AMEBA SERVICES

Ameba Pigg

CONTENTS

¡  Apache Ignite

¡  Feed your data

¡  Log Aggregation with Apache Flume

¡  Integration with Apache Ignite

¡  Streaming Data with Apache Kafka

¡  Data Pipeline with Kafka and Ignite: Example

APACHE IGNITE

¡  “High-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash-based technologies.”

¡  High performance, unlimited scalability and resiliency

¡  High-performance transactions and fast analytics

¡  Hadoop Acceleration, Apache Spark

¡  Apache project

https://ignite.apache.org/

MAKING APACHE IGNITE A DATA PROCESSING HUB

¡  Question: How to feed data?

¡  A simple solution: Create a client node

MAKING APACHE IGNITE A DATA PROCESSING HUB

¡  Question: How to feed data?

¡  A simple solution: Create a client node

¡  Is it reliable?

¡  Does it scale?

¡  Ignite-only solution?

¡  Does it keep your operational costs low?

MAKING APACHE IGNITE A DATA PROCESSING HUB

¡  Question: How to feed data?

¡  A simple solution: Create a client node

¡  Is it reliable?

¡  Does it scale?

¡  Ignite-only solution?

¡  Does it keep your operational costs low?

LOG AGGREGATION WITH APACHE FLUME

LOG AGGREGATION WITH APACHE FLUME

¡  Flume

¡  “Distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.”

¡  Scalable

¡  Flexible

¡  Robust and fault tolerant

¡  Declarative configuration

¡  Apache project

DATA FLOW IN FLUME

Source Sink

Agent

Channel Incoming data

to another Agent or Destination

DATA FLOW IN FLUME (REPLICATION/MULTIPLEXING)

Source Sink

Agent

Channel Incoming data

Sink Channel Channel Selector

DATA FLOW IN FLUME (RELIABILITY)

¡  No data is lost (configurable)

Source Sink

Agent

Channel Incoming data

Source tx Sink tx

LOG TRANSFER AT AMEBA

Ameba Service � Aggregator

Aggregator

Aggregator

Monitoring Recommender

System

Elastic Search

Hadoop Batch processing

HBase

Stream Processing (Onix)

Stream Processing (HBaseSink)

Ameba Service �

Ameba Service �

LOG TRANSFER AT AMEBA

¡  Web Hosts

¡  More than 1600

¡  Size

¡  5.0 TB/day (raw)

¡  Traffic at peak

¡  160Mbps (compressed)

IGNITE SINK

¡  Reads Flume events from a channel

¡  With a user-implemented pluggable transformer converts them into cacheable entries

¡  Adding it requires no modification to the existing architecture

FLUME ⇒ IGNITE (1)

Source Ignite Sink

Agent

Channel Incoming data new connection

FLUME ⇒ IGNITE (2)

Source Ignite Sink

Agent

Channel Incoming data

Sink tx

start tx

FLUME ⇒ IGNITE (3)

Source Ignite Sink

Agent

Channel Incoming data

Sink tx

take event send events

ENABLING FLUME SINK

¡  Steps

1.  Implement EventTransformer

¡  convert Flume events into cacheable entries (java.util.Map<K, V>)

2.  Put transformer’s jar to ${FLUME_HOME}/plugins.d/ignite/lib

3.  Put IgniteSink and Ignite core jar files to ${FLUME_HOME}/plugins.d/ignite/libext

4.  Set up a Flume agent

¡  Sink setup

a1.sinks.k1.type = org.apache.ignite.stream.flume.IgniteSink a1.sinks.k1.igniteCfg = /some-path/ignite.xml a1.sinks.k1.cacheName = testCache a1.sinks.k1.eventTransformer = my.company.MyEventTransformer a1.sinks.k1.batchSize = 100

FLUME SINKS

¡  HDFS

¡  THRIFT

¡  AVRO

¡  HBASE

¡  ElasticSearch

¡  IRC

¡  IGNITE

APACHE FLUME & APACHE IGNITE

¡  If you do data aggregation with Flume

¡  Adding an Ignite cluster is as simple as writing a simple data transformer and deploying a new Flume agent

¡  If you store your data (and do computations) in Ignite

¡  Improving data injection becomes easy with Flume sink

¡  Combining Apache Flume and Ignite makes/keeps your data pipeline (both aggregation and processing) ¡  Scalable

¡  Reliable

¡  Highly-Performant

STREAMING DATA WITH APACHE KAFKA

APACHE KAFKA

“Publish-subscribe messaging rethought as a distributed commit log”

¡  Low latency

¡  High Throughput

¡  Partitioned and Replicated

¡  Kafka is an essential component of any data pipeline today

http://kafka.apache.org/

APACHE KAFKA

¡  Messages are grouped in topics

¡  Each partition is a log

¡  Each partition is managed by a broker (when replicated, one broker is the partition leader)

¡  Producers & consumers (consumer groups)

¡  Used for

¡  Log aggregation

¡  Activity tracking

¡  Monitoring

¡  Stream processing

http://kafka.apache.org/documentation.html

KAFKA CONNECT

¡  Designed for large scale stream data integration using Kafka

¡  Provides an abstraction from communication with your Kafka cluster

¡  Offset management

¡  Delivery semantics

¡  Fault tolerance

¡  Monitoring, etc.

¡  Worker (scalability & fault tolerance)

¡  Connector (task config)

¡  Task (thread)

¡  Standalone & Distributed execution models http://www.confluent.io/blog/apache-kafka-0.9-is-released

INGESTING DATA STREAMS

¡  Two ways

¡  Kafka Streamer

¡  Sink Connector

SQL queries Distributed closures Transactions

Con

nect

ETL

STREAMING VIA SINK CONNECTOR

¡  Configure your connector

¡  Configure Kafka Connect worker

¡  Start your connector

# connectorname=my-ignite-connectorconnector.class=IgniteSinkConnectortasks.max=2topics=someTopic1,someTopic2# cachecacheName=myCachecacheAllowOverwrite=trueigniteCfg=/some-path/ignite.xml

$ bin/connect-standalone.sh myconfig/connect-standalone.properties myconfig/ignite-connector.properties

STREAMING VIA SINK CONNECTOR

¡  Easy data pipeline

¡  Records from Kafka are written to Ignite grid via high-performance IgniteDataStreamer

¡  At-least-once delivery guarantee

¡  As of 1.6, start a new connector to write to a different cache

a b c d e

0 1 2 … Kafka offsets

a.key, a.val b.key, b.val …

a2 b2 c2 d2 e2

INGESTING DATA STREAMS

¡  Bi-directional streaming

SQL queries Distributed closures Transactions

Con

nect

Events Continuous queries

Con

nect

Si

nk

Sour

ce

STREAMING BACK TO KAFKA

¡  Listening to cache events

¡  PUT

¡  READ

¡  REMOVED

¡  EXPIRED, etc.

¡  Remote filtering can be enabled

¡  Kafka Connect offsets are ignored

¡  Currently, no delivery guarantees

evt1

evt2

evt3 as records

ENABLING SOURCE CONNECTOR

¡  Configure your connector

¡  Define a remote filter if needed cacheFilterCls=MyCacheEventFilter

¡  Make sure that event listening is enabled on the server nodes

¡  Configure Kafka Connect worker

¡  Start your connector

#connector name=ignite-src-connector connector.class=org.apache.ignite.stream.kafka.connect.IgniteSourceConnector tasks.max=2 #topics, events topicNames=test cacheEvts=put,removed #cache cacheName=myCache igniteCfg=myconfig/ignite.xml

key.converter=org.apache.kafka.connect.storage.StringConverter value.converter=org.apache.ignite.stream.kafka.connect.serialization.CacheEventConverter

APACHE KAFKA & APACHE IGNITE

¡  If you do data streaming with Kafka

¡  Adding an Ignite cluster is as simple as writing a configuration file (and creating a filter if you need it for source)

¡  If you store your data (and do computations) in Ignite

¡  Improving data injection and listening for events on data becomes easy with Kafka Connectors

¡  Combining Apache Kafka and Ignite makes/keeps your data pipeline

¡  Scalable

¡  Reliable

¡  Highly-Performant

¡  Covers a wide range of ETL contexts

DATA PIPELINE WITH KAFKA AND IGNITE EXAMPLE

DATA PIPELINE WITH KAFKA AND IGNITE

¡  Requirements

¡  instant processing and analysis

¡  scalable and resilient to failures

¡  low latency

¡  high throughput

¡  flexibility

DATA PIPELINE WITH KAFKA AND IGNITE

¡  Filter and aggregate events

data Flume

filter/transform

data

data

slow down on heavy loads

more channels/layers

DATA PIPELINE WITH KAFKA AND IGNITE

data

filter transform

etc.

•  Parsimonious resource use •  Replay enabled •  More operations on streams •  Flexibility

Other sources

DATA PIPELINE WITH KAFKA AND IGNITE

¡  Filter and aggregate events

¡  Store events

¡  Notify about updates on aggregates

data

filter transform

etc.

Connectors

DATA PIPELINE WITH KAFKA AND IGNITE

¡  Filter and aggregate events

¡  Store events

¡  Notify about updates on aggregates

data

filter transform

etc.

Connectors

DATA PIPELINE WITH KAFKA AND IGNITE

¡  Improving ads delivery

clicks impressions

ads

Ads delivery

Ads recommender

storage/ computation

Image storage

data & computation in one place

DATA PIPELINE WITH KAFKA AND IGNITE

¡  Improving ads delivery ¡  Better network utilization and reliability

clicks impressions

ads

Ads delivery

Ads recommender

storage/ computation

Image storage

Anomaly detection

OTHER INTEGRATIONS

OTHER COMPLETED INTEGRATIONS

¡  CAMEL

¡  MQTT

¡  STORM

¡  FLINK SINK

¡  TWITTER

THE END