detecting events on the web in real time with java, kafka and zookeeper - james stanier

Dr. James Stanier | Head of Analytics | Brandwatch.com | [email protected]

Coming Up/

© 2014 Brandwatch | www.brandwatch.com 2

•  Me, Brandwatch and new problems

•  Apache Kafka

•  Processing data in Java

•  Distributing work with Zookeeper

•  Managing state


Who?


Dr. James Stanier Head of Analytics, Brandwatch @jstanier | [email protected]


Brandwatch


Where we are/

•  Brighton

•  New York

•  San Francisco

•  Berlin

•  Stuttgart

Crawl Store and Index

What Brandwatch does/


Analyse

3

Present

4

•  Crawl 70M+ sites including key social networks

•  27 languages

•  Powerful search operators

•  20Bn + indexed URLs

•  Years of historical data

•  Automated topic & sentiment analysis in all 27 languages

•  Automate common tasks including alerts

•  Advanced analytics modules

•  Automatic categorisation with rules

•  Custom dashboards

•  Reporting & alerts


Brandwatch Analytics

10

Data/ Presentation


Data/ Aggregation


Data/ Classification


Data/ Not just top level metrics


Development/ What do we use?


Data/ The numbers

•  50+ Java Web Crawlers

•  10+ Historical crawlers for new queries

•  Twitter via GNIP (now Twitter)

•  70M+ query matches per day


The speed of social


A new challenge

21

The challenge/ The signal from the noise


•  100K+ user queries

•  70M+ mentions per day

•  Polling the database for mentions would take

8hrs for one pass

The challenge/ at scale


Crawler 1

Crawler 2

Crawler n-1

Crawler N

Kafka Cluster

Signals Processing

cluster

Signals handler

JVM

DB

Mentions

Signals

Signals

Mentions

The Problem/ How we handled it…


Kafka

Step 1/ Kafka


Crawler 1

Crawler 2

Crawler n-1

Crawler N

Kafka Cluster Mentions


Kafka/ What is it?

•  Apache Kafka is a publish-subscribe messaging

system rethought as a distributed commit log

•  Apache top level project November 2013

•  Started at LinkedIn


Kafka/ is…

•  Fast: hundreds of MBs read/write per second

from thousands of clients

•  Scalable: clustered, partitioned over many

machines, expanded without downtime

•  Durable: messages persisted to disk and

replicated in cluster


Kafka/ Terminology

•  Kafka maintains feeds of messages

called topics

•  Programs that publish messages are

called producers

•  Programs that subscribe to messages

are called consumers

•  Kafka is a cluster of servers called brokers

Kafka/ How it’s used...


Producer Producer Producer

Consumer Consumer Consumer

Kafka Cluster

Kafka/ Written to disk?


http://q.acm.org/detail.cfm?id=1563874

Kafka/ Bending, not breaking


http://engineering.gnip.com/tag/kafka/

Kafka/ The anatomy of a topic


0 2 1 5 3 4 6

0 2 1 3 4

0 2 1 5 3 4

Partition 0

Partition 1

Partition 2

Old New

Writes


Kafka/ Warning: ordering

Kafka guarantees a total ordering per partition,

not per whole topic


Kafka/ Try it out! > tar -xzf kafka_2.9.2-0.8.1.tgz!

> cd kafka_2.9.2-0.8.1!

> bin/zookeeper-server-start.sh config/zookeeper.properties!

> bin/kafka-server-start.sh config/server.properties!

!


Kafka/ Try it out! > bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test!

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test!

Hello JAX!!

> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning!

Hello JAX!!


Kafka/ With Java <dependency>!

!<groupId>org.apache.kafka</groupId>!

!<artifactId>kafka_2.10</artifactId>!

!<version>0.8.1</version>!

</dependency>!


Kafka/ With Java Properties props = new Properties();!

props.put("metadata.broker.list", "broker1:9092,broker2:9092");!

props.put("serializer.class", "kafka.serializer.StringEncoder");!

props.put("partitioner.class", "example.producer.SimplePartitioner");!

props.put("request.required.acks", "1");!

ProducerConfig config = new ProducerConfig(props);!

Producer<String, String> producer = new Producer<String, String>(config);!


Kafka/ Partitioning

public class SimplePartitioner implements Partitioner<String> {!

public SimplePartitioner (VerifiableProperties props) {!

!

public int partition(String key, int numberOfPartitions) {!

return md5hash(key) % numberOfPartitions;!

}!

}!


Kafka/ Sending from the crawlers String message = toJson(...);!

KeyedMessage<String, String> message = new!

!KeyedMessage<String, String>("query.mentions", !queryId, message);!

producer.send(message);!

Step 1/ Done


Crawler 1

Crawler 2

Crawler n-1

Crawler N



Processing

42

Processing/ What’s happening now?

Step 2.1/ One processing JVM


Crawler 1

Crawler 2

Crawler n-1

Crawler N


Signals processor

Processing/ A wild tweet appears!


Mention date: 14/10/2014 6:10PM pageType: twitter author: @javadude hashtags: [#jaxlondon, #amazingtalk, #greatshoes] mentionedTweeters: [@jstanier] text: “@jstanier is at #jaxlondon tonight #amazingtalk #greatshoes”

Processing/ Storing hashtags


Map<Date, Multiset<String>>!

!

Initialise with the last 24 hours




!

Mention date: 14/10/2014 6:10PM hashtags: [#jaxlondon, #amazingtalk, #greatshoes]




!

Mention date: 14/10/2014 6:10PM hashtags: [#jaxlondon, #amazingtalk, #greatshoes] add(“#jaxlondon”)!add(“#amazingtalk”)!add(“#greatshoes”)!


Processing/ Cycling the buckets @Scheduled(cron = "0 0 * * * *")!

public void cycleBuckets() {!

Date oldest = buckets.lastKey();!

removeBucket(oldest);!

DateTime newest = new!

! ! !DateTime(buckets.firstKey());!

addBucket(newest.plusHours(1).toDate());!

}!

Processing/ Detecting spikes


•  At regular intervals •  For each hashtag

•  Convert to a timeseries [5, …. 1002, 5499] •  Use our super secret detection algorithm •  Give a score to it

•  If score > threshold, it’s interesting •  Send it on a new Kafka topic

Processing/ What we just did


#hashtag data

model

Processing/ But we also track…


author data

model

sentiment data

model

page type data

model

link share data

model

#hashtag data

model

country data

model

volume data

model

Processing/ …for one query


“JAX London” query

author data

model

sentiment data

model

page type data

model

link share data

model

#hashtag data

model

country data

model

volume data

model

Processing/ 100K+ queries and rising



Processing/ We need more JVMs

But how do we share the workload?


Distribution of work

© 2014 Brandwatch | www.brandwatch.com

Crawler 1

Crawler 2

Crawler n-1

Crawler N

Kafka Cluster

Signals Processing

cluster

Mentions Signals

Mentions

Step 2.2/ A cluster of processing JVMs

© 2014 Brandwatch | www.brandwatch.com

Signals Processing

cluster

Distribution/ An atomic unit of work

?


Distribution/ Leader election

A way of deciding who is the leader for a task in a group of distributed nodes


Distribution/ Zookeeper

A way of coordinating and managing distributed applications


Zookeeper/ It’s like a file system

/brandwatch

/feature_2 /feature_1


Zookeeper/ At the command line


Distribution/ In Java

http://curator.apache.org/curator-framework


Distribution/ Recipes


Distribution/ Instantiating Curator !

CuratorFrameworkFactory!

.builder()!

.connectString(zkQuorum)!

.namespace(namespace)!

.build()!

! ! ! !.start();!


Distribution/ Offering jobs

/brandwatch

/signals

/queries

/1268589 /15846

Manager JVM

DB


Distribution/ Adding nodes !

public void createZNode(String queryId) {!

try {!

client.create().forPath(ZK_NODE_PREFIX + queryId);!

} catch (NodeExistsException e) {!

log.debug("Node {} was already created.”, queryId);!

}!

}!


Distribution/ Deleting nodes !

public void removeZNode(String queryId) {!

try {!

client.delete().forPath(ZK_NODE_PREFIX + queryId);!

} catch (NoNodeException e) {!

log.debug("Node {} was already deleted.”, queryId);!

}!

}!


Distribution/ Leader election 101

/brandwatch

/signals

/queries

/1268589 /15846

Processing JVM 1

Processing JVM 2

Processing JVM 3


Distribution/ Leader election 101

/brandwatch

/signals

/queries

/1268589 /15846

1 2 3

Processing JVM 1

Processing JVM 2

Processing JVM 3


Distribution/ The leader dies

/brandwatch

/signals

/queries

/1268589 /15846

2 3

Processing JVM

Processing JVM


Distribution/ The dead rises again

/brandwatch

/signals

/queries

/1268589 /15846

2 3 4

Processing JVM

Processing JVM

Processing JVM


Distribution/ Curator: LeaderLatch !

public LeaderLatch(CuratorFramework client, String latchPath)!

Parameters:!

!

client - the client!

latchPath - the path for this leadership group!


Distribution/ LeaderLatch recipe public class WorkerManager implements PathChildrenCacheListener {!

!

private Map<Integer, LeaderLatch> leaderLatches = newHashMap();!

!

@Override!

public void childEvent(CuratorFramework client, !

PathChildrenCacheEvent event) {!

// Handle adds and removes here!!

}!

}!


Distribution/ Curator: Starting up @PostConstruct!

public void initialise() throws Exception {!

List<ChildData> currentData = newArrayList(initialisePathChildrenCache());!

log.info("Pre creating workers for {} existing queries", currentData.size());!

for (ChildData childData : currentData) {!

int queryId = parseQueryIdFromPath(childData.getPath());!

startLeaderElection(queryId);!

}!

}!


Distribution/ Curator: PathChildrenCache private List<ChildData> initialisePathChildrenCache() throws Exception {!

pathChildrenCache.start(StartMode.BUILD_INITIAL_CACHE);!

pathChildrenCache.getListenable().addListener(this);!

List<ChildData> currentData = pathChildrenCache.getCurrentData();!

return currentData;!

}!


Distribution/ Curator: Adding a node @Override!

public void childEvent(CuratorFramework client, PathChildrenCacheEvent event) { !

ChildData childData = event.getData();!

switch (event.getType()) {!

case CHILD_ADDED:!

queryId = parseQueryIdFromPath(childData.getPath());!

if (!haveLeaderLatchForQuery(queryId)) {!

startLeaderElection(queryId);!

}!

break;!


Distribution/ Curator: Deleting a node // Continued...!

case CHILD_REMOVED:!

queryId = parseQueryIdFromPath(childData.getPath());!

removeLeaderLatchForQuery(queryId);!

break;!

default:!

break;!

}!

}!


We are processing long running jobs

What about workers getting overloaded?

Distribution/ Almost there?


1.  Take leadership

2.  Hit max queries?

a.  No – go to 3

b.  Yes – give up leadership, try again

3.  Start working

Distribution/ After leader election


Actually, no…

Distribution/ Now we’re almost there?


Distribution/ Infinite election

Processing JVM

Processing JVM Processing JVM

At capacity! At capacity!

At capacity!

Elected for 1328

Elected for 1328

Elected for 1328


Distribution/ Solution

Processing JVM


At capacity!

Elected for 1328

Elected for 1328

Refused 1328


Distribution/ Solution

Processing JVM


At capacity! At capacity!

At capacity!

Elected for 1328

Elected for 1328

Elected for 1328

Refused 1328

Refused 1328

Refused 1328


State


State/ CAP theorem

Availability

CA AP

CP


If one worker dies, we want the other to pick up where it left off

Regular snapshotting to HBase

State/ Snapshotting of worker data


Serialise and compress using Kryo

~ 0.5MB per query, but a lot are very small

State/ Serialisation and compression


Crawler 1

Crawler 2

Crawler n-1

Crawler N

Kafka Cluster

Signals Processing

cluster

Signals handler

JVM

DB

Mentions

Signals

Signals

Mentions

Step 2.2/ Done!


Monitoring


Monitoring/ Statsd and Graphite


Closing remarks


Now •  Smarter alerts (email, push, in-browser) •  Monitoring crises/events as they happen

Underway •  Automatic clustering of spikes into events •  Historical analysis of trends

Summary/ Using this architecture

Say hello/


@brandwatch | @jstanier

www.brandwatch.com

UK: +44 (0)1273 358 635

[email protected]


Q&A

detecting events on the web in real time with java, kafka and zookeeper - james stanier

Presentations & Public Speaking

kafka terminology kafka

consumers kafka

new challenge

data aggregation

data classification

data presentation

new queries twitter

java web crawlers