Download - Real-time Data Processing
Real-Time Data ProcessingEmerging Business Meetup - 07/24/13Bryan Warner @ Traackr
About Me● Bryan Warner - Engineer @ Traackr
● Primary background is in Java○ Breaking into Scala development this past year
● Interested in search, data scalability, and distributed computing
About Traackr● Influencer search engine
○ Platform for discovering and engaging online individuals who matter
● We track content and metrics for our database of influential people○ Both in RT and daily processes
● Some of our back-end stack includes: ElasticSearch, MongoDb, Java/Spring, Scala/Akka, etc.
● Looking for developers to use our API!
Overview● Review Traackr's use case for real-time data
processing
● Technical solution we decided on
● Questions
Traackr Use Case1. Real-time content stream for a targeted group of
influencers within our platform
a. Primarily to show real-time tweets via our Twitter data provider (GNIP)
2. On-demand content tracking and searching for new influencers
a. Users can add up to a hundred people at onceb. Expect that new influencer content is searchable
near real-time
Traackr Use CaseData Processing Requirements
1. Incoming data is not lost
2. Data needs to be analyzed and enriched
3. Each type of data has its own processing component* Blog Posts, Tweets, Videos, Images, etc.
4. Components should be configurable for maximum throughput!
5. Components should act like small building blocks
Bird's Eye View
Tracking App
MongoDb
Initial Persist
Content "Enrichment" Pipeline
RabbitMQ Broker
Queue
ElasticSearch
GNIP Listener App
"Post" Payload
Queue Listener
Make content searchable
Content Pipeline● Apache Camel (http://camel.apache.org/)
○ Integration framework based on Enterprise Integration patterns (EIP)
● Flexible route building○ Supports direct and asynchronous components○ Integrates with DI frameworks (e.g. Spring, Guice)○ Tons of native support for various transports (http,
jms, amqp, tcp, imap, etc.)
● Good support for unit testing○ org.apache.camel.component.mock.MockEndpoint
Content PipelineQueue
Queue Listener
Search Indexer
Tweet Processor
Blog Processor
Image Processor
Routing Filter
ROUTE
Content PipelineInitial Approach:
● Route(s) live within a CamelContext in your JVM● Initial route is utilizing direct components (serial)
from(<queue.uri>).routeId("my-route").choice()
.when(simple("${in.body.isTweet()}")) .to("bean:languageAnalyzer?method=detectLanguage") .to("bean:tweetAnalyzer?method=extractMentions")
.when(simple("${in.body.isBlog()}")).to("bean:httpService?method=fetchFullContent")
.to("bean:languageAnalyzer?method=detectLanguage") .otherwise() .to("bean:imageAnalyzer?method=categorizeImage")
.end()
.to("bean:searchService?method=indexContent");
But there's a throughput problem...
Content PipelineTWEET
TWEET
IMAGE
TWEET
TWEET
BLOG
TWEET
TWEET
TWEET
BLOG
TWEET
TWEET
● If Tweets come into the system at 5/sec, then Tweet processing rate has to be >= 5/sec
● If a blog post takes 5 seconds to process (on average)...
● And an image takes 30 seconds to process (on average)...
then...
Queue
HEAD
Content PipelineExpanded Approach:
● Utilize Seda Components (http://camel.apache.org/seda.html)○ Underlying Thread pool with BlockingQueue
from(<queue.uri>).routeId("my-route").choice()
.when(simple("${in.body.isTweet()}")).to("seda:tweetEnricher") .when(simple("${in.body.isBlog()}")).to("seda:blogEnricher") .otherwise().to("seda:imageEnricher")
.end();
from("seda:tweetEnricher?concurrentConsumers=10").routeId("tweet-route").to("bean:languageAnalyzer?method=detectLanguage").to("bean:tweetAnalyzer?method=extractMentions").to("seda:searchService")
from("seda:blogEnricher?concurrentConsumers=2").routeId("blog-route") ...from("seda:imageEnricher?concurrentConsumers=2").routeId("img-route") ...
// Routes re-join from("seda:searchService?concurrentConsumers=X").routeId("s-indexer") .to("bean:searchService?method=indexContent");
Content PipelineQueue
Queue Listener
Search Indexer
Tweet Processor
Blog Processor
Image Processor
ROUTE
Routing Filter
ThreadPool + BlockingQueue
Content PipelineCaveats:
● No visibility into SEDA's thread pool state (e.g. how many objects on its internal queue?)
● If VM crashes, those payloads on the SEDA thread pool blocking queue are lost
● Our route is assuming that each payload consists of only one message○ In reality, our payloads are a mix of different post types ... how to
handle this efficiently?
Content PipelineFinal Solution:
from(<queue.uri>).routeId("my-route").split().method("payloadSplitterService", "splitMessage").choice()
.when(header("enrichTweets").isEqualTo(true)).to(<queue.uri.tweet>) .when(header("enrichBlogs").isEqualTo(true)).to(<queue.uri.blogs>)
.otherwise().to("<queue.uri.images>").end();
from(<queue.uri.tweet>).routeId("queue-in-tweet-route") .to("seda:tweetEnricher?timeout=0");
from("seda:tweetEnricher?concurrentConsumers=10&size=0&blockWhenFull=true").routeId("tweet-route")
.to("bean:languageAnalyzer?method=detectLanguage") .to("bean:tweetAnalyzer?method=extractMentions") .to("seda:searchService")
from("seda:searchService?concurrentConsumers=X").routeId("s-indexer") .to("bean:searchService?method=indexContent");
Questions