aws real-time event processing
TRANSCRIPT
April 21, 2015
Seattle
AWS Big Data Platform
Agenda Overview
10:00 AM Registration
10:30 AM Introduction to Big Data @ AWS
12:00 PM Lunch + Registration for Technical Sessions
12:30 PM Use Case Technical Deep Dive Sessions • Data Collection and Storage
• Real-time Event Processing
• Analytics
Collect Process Analyze Store
Data Collection and Storage
Data Processing
Data Analysis
Event Processing
Primitive Patterns
S3 Kinesis DynamoDB RDS (Aurora)
MySQL
AWS Lambda
KCL Apps EMR Redshi?
Machine Learning
Real-Time Event Processing
• Examples:
Processing framework
Two main processing patterns
Real-time event processing frameworks
Kinesis Client Library
AWS Lambda
Amazon KCL
Shard 1
Shard 2
Shard 3
Shard n
Shard 4
KCL Worker 1
KCL Worker 2
EC2 Instance
KCL Worker 3
KCL Worker 4
EC2 Instance
KCL Worker n
EC2 Instance
Kinesis
Kinesis Client Library (KCL)
KCL Design Components
KCL restarts the processing of the shard at the last known processed record if a worker fails
Processing with Kinesis Client Library • Connects to the stream and enumerates the shards
• Instantiates a record processor for every shard it manages
• Checkpoints processed records in Amazon DynamoDB
• Balances shard-worker associations when the worker instance count changes
• Balances shard-worker associations when shards are split or merged
Best practices for KCL applications
Amazon Kinesis Connector
S3 Dynamo DB Redshift
Kinesis
Amazon Kinesis connector application
Connector Pipeline
Transformed
Filtered
Buffered
Emitted
Incoming Records
Outgoing to Endpoints
Real-time Monitoring dashboard with KCL
Amazon Kinesis
Kinesis-enabled Application
Producer on Amazon EC2
Amazon DynamoDB
Dashboard on Amazon EC2
2 sec sliding-window analysis over streaming clickstream data
Monitoring Demo Kinesis Client Library
AWS Lambda
Event-Driven Compute in the Cloud
No Infrastructure to Manage
Automatic Scaling
Bring your own code
Fine-grained pricing
Free Tier 1M requests and 400,000 GB-s of compute.
Every month, every customer.
Never pay for idle.
Data Triggers: Amazon S3
Amazon S3 Bucket Events AWS Lambda
Original image Thumbnailed image
1
2
3
Data Triggers: Amazon DynamoDB
AWS Lambda Amazon DynamoDB Table and Stream
Send Amazon SNS notifications
Update another table
Calling Lambda Functions
Writing Lambda Functions
How can you use these features?
“I want to send customized messages to
different users”
SNS + Lambda
“I want to send an offer when a user runs out of lives in
my game”
Amazon Cognito + Lambda + SNS
“I want to transform the
records in a click stream or an IoT
data stream”
Amazon Kinesis + Lambda
Real-Time Alerting Demo AWS Lambda
Stream Processing Apache Spark Apache Storm Amazon EMR
Read Data Directly into Hive, Pig, Streaming and Cascading Real time sources into Batch Oriented Systems Multi-Application Support & Check-pointing
Amazon EMR integration
CREATE TABLE call_data_records ( start_time bigint, end_time bigint, phone_number STRING, carrier STRING, recorded_duration bigint, calculated_duration bigint, lat double, long double ) ROW FORMAT DELIMITED FIELDS TERMINATED BY "," STORED BY 'com.amazon.emr.kinesis.hive.KinesisStorageHandler' TBLPROPERTIES("kinesis.stream.name"=”MyTestStream");
Amazon EMR integration: Hive
DStream
RDD@T1 RDD@T2
Messages
Receiver
Spark Streaming – Basic concepts
http://spark.apache.org/docs/latest/streaming-kinesis-integration.html
Spark Streaming
Processing Amazon Kinesis streams
Amazon Kinesis
Spark-Streaming
Weblog Demo Kinesis + Spark Streaming
Storm
Apache Storm: Basic Concepts
https://github.com/awslabs/kinesis-storm-spout
Launches Workers
Storm architecture
Master Node
Cluster Coordination
Worker Processes
Worker
Nimbus
Zookeeper
Zookeeper
Zookeeper
Supervisor
Supervisor
Supervisor
Supervisor
Worker
Worker
Worker
Real-time: Event-based processing
Kinesis Storm Spout
Producer Amazon Kinesis
Apache Storm
ElastiCache (Redis) Node.js Client
(D3)
http://blogs.aws.amazon.com/bigdata/post/Tx36LYSCY2R0A9B/Implement-a-Real-time-Sliding-Window-Application-Using-Amazon-Kinesis-and-Apache
Thank You