© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Warren Paull, Solution Architect, AWSPatricio Rocca, Chief Technology Officer, Jampp

July 2016

Getting Started withAmazon Kinesis

What to expect from this session

• Streaming scenarios• Amazon Kinesis Streams overview• Amazon Kinesis Firehose overview • Firehose experience for Amazon S3 and Amazon Redshift

• Jampp – Our Journey with Amazon Kinesis

Streaming Data Use Cases

AcceleratedIngest-

Transform-Load

Continual Metric

Generation

ResponsiveData

Analysis

1 2 3

Amazon Kinesis Streams

Build your own custom applications that

process or analyze streaming data

Amazon Kinesis Firehose

Easily load massive volumes of streaming data into Amazon S3 and Amazon Redshift

Amazon Kinesis Analytics

Easily analyze data streams using

standard SQL queries

Amazon Kinesis: Streaming data made easyServices make it easy to capture, deliver, and process streams on AWS

In Preview

Amazon Kinesis: Streaming data done the AWS wayMakes it easy to capture, deliver, and process real-time data streams

Pay as you go, no upfront costs

Elastically scalable

Right services for your specific use cases

Real-time latencies

Easy to provision, deploy, and manage

Amazon Kinesis StreamsBuild your own data streaming applications

Easy administration: Simply create a new stream and set the desired level of capacity with shards. Scale to match your data throughput rate and volume. Build real-time applications: Perform continual processing on streaming big data using Amazon Kinesis Client Library (KCL), Apache Spark/Storm, AWS Lambda, and more. Low cost: Cost-efficient for workloads of any scale.

Reading and Writing with Streams

AWS SDK

LOG4J

Flume

Fluentd

Get* APIs

Kinesis Client Library+

Connector Library

Apache Storm

Amazon Elastic MapReduce

Sending Consuming

AWS Mobile SDK

KinesisProducerLibrary

AWS Lambda

Apache Spark

Real-time streaming data ingestion

Custom-built streaming applications

Inexpensive: $0.014 per 1,000,000 PUT payload units

Amazon Kinesis StreamsManaged service for real-time processing

We listened to our customers…

Amazon Kinesis FirehoseLoad massive volumes of streaming data into Amazon S3 and Amazon Redshift

Zero administration: Capture and deliver streaming data into Amazon S3, Amazon Redshift, and other destinations without writing an application or managing infrastructure.

Direct-to-data store integration: Batch, compress, and encrypt streaming data for delivery into data destinations in as little as 60 secs using simple configurations.

Seamless elasticity: Seamlessly scales to match data throughput w/o intervention.

Capture and submit streaming data to Firehose

Firehose loads streaming data continuously into S3 and

Amazon Redshift

Analyze streaming data using your favorite BI tools

AWS Platform SDKs Mobile SDKs Kinesis Agent AWS IoT

Amazon S3

Amazon Redshift

• Send data from IT infra, mobile devices, sensors • Integrated with AWS SDK, agents, and AWS IoT• Batch, compress, and encrypt data before loads• Loads data into Amazon Redshift tables by using

the COPY command

• Pay-as-you-go: 3.5 cents / GB transferredAmazon Kinesis Firehose

Capture IT and app logs, device and sensor data, and more Enable near-real time analytics using existing tools

AmazonElasticSearch

1. Delivery stream: The underlying entity of Firehose. Use Firehose bycreating a delivery stream to a specified destination and send data to it.• You do not have to create a stream or provision shards.• You do not have to specify partition keys.

2. Records: The data producer sends data blobs as large as 1,000 KB to adelivery stream. That data blob is called a record.

3. Data Producers: Producers send records to a delivery stream. Forexample, a web server that sends log data to a delivery stream is a dataproducer.

Amazon Kinesis Firehose Three simple concepts

Amazon Kinesis Firehose console experience Unified console experience for Firehose and Streams

Amazon Kinesis Firehose console (S3) Create fully managed resources for delivery without building an app

Amazon Kinesis Firehose console (S3) Configure data delivery options simply using the console

Amazon Kinesis Firehose console (Amazon Redshift)Configure data delivery to Amazon Redshift simply using the console

Amazon Kinesis agentSoftware agent makes submitting data easy

• Monitors files and sends new data records to your delivery stream• Handles file rotation, check pointing, and retry upon failures• Preprocessing capabilities such as format conversion and log parsing• Delivers all data in a reliable, timely, and simple manner• Emits Amazon CloudWatch metrics to help you better monitor and

troubleshoot the streaming process• Supported on Amazon Linux AMI with version 2015.09 or later, or Red Hat

Enterprise Linux version 7 or later; install on Linux-based server environments such as web servers, front ends, log servers, and more

• Also enabled for Streams

Amazon Kinesis Firehose or Amazon Kinesis Streams?

Amazon Kinesis Streams is a service for workloads that requirescustom processing, per incoming record, with sub-1-secondprocessing latency, and a choice of stream processing frameworks.

Amazon Kinesis Firehose is a service for workloads that requirezero administration, ability to use existing analytics tools basedon S3, Amazon Redshift and Amazon Elasticsearch with datalatency of 60 seconds or higher.

Amazon Kinesis Analytics

Amazon Kinesis AnalyticsAnalyze data streams continuously with standard SQL

Apply SQL on streams: Easily connect to data streams and apply existing SQL skills.

Build real-time applications: Perform continual processing on streaming big data with sub-second processing latencies.

Scale elastically: Elastically scales to match data throughput without any operator intervention.

Connect to Amazon Kinesis streams, Firehose delivery

streams

Run standard SQL queries against data streams

Amazon Kinesis Analytics can send processed data to analytics tools so you can create alerts and respond in real time

In Preview

Patricio Rocca | July 2016

Jampp Padawans journey with Kinesis

About Jampp

We are a tech company that helps companies grow their mobile business by driving engaged users to their apps

Machine learning

Post-install event optimisation

Dynamic Product Ads and Segments

Data Science

Programmatic Buying

We are a team of 70 people, 30%in the engineering team.

Located in 6 cities across the US, Latin America, Europe and Africa

About Real-Time Bidding

• Ad Impressions are available through an Exchange

• Demand Platforms have to bid in less the 100 ms

• The highest bid wins the impressions and shows the ad!

We do this 220,000 times per second

Bid

Real-Time Bidding Workflow

Auction Win

Exchange Exchange

Publisher Publisher

Jampp Bidder

Jampp Machine Learning

Jampp Engagement

Segments Builder

IMPRESSION ;-)PLACEHOLDER

Real-Time Tracking Workflow

In-App Event

TrackingPlatform

Jampp Client

Application

Jampp NodeJSTrackingPlatform

• Build a retargeting platform that generates groups of users based on their in-app activity and a look-a-like machine learning model

• Process and enrich in-app events in less than 5 minutes to target users when they become dormant

• Build a scale-on-demand platform that lets our business grow without pain• Increase the platform’s monitoring, logging and alerting capabilities

Also…

Non tech people should be able to query granular data and aggregate it over large periods

Business Challenges

• 700M events/300GB per day

• 1500% in-app events growthYoY

• Growth factor peaks out of tech team control since it depends on the sales team pacing ;-)

Data Scale

Start your engines!

The Phantom Menace (initial architecture)

Cost Savings

• Kafka supports higher throughput and lower latency and there are tons of successful implementation cases but few are managing similar volume like Jampp

• EBS allocation per topic was hard to size correctly, tune and scale “on demand”

• Kafka maintainability required dedicated manpower

• Kafka security configuration was not flexible enough to add secured producers outside of the VPC

$ 2,848 $ 936

Tempted by the Dark Side

A New Hope (final architecture)

• Invested several days picking the partition key for evenly distributing data across shards

• Encoding protocol matters! Performed several benchmarks and MessagePack offered the best trade off between compression and serialization speed factor

Jedi Trial I

• Write/Read Batching to reduce the HTTPS protocol overhead and costs

• Exponential backoff + Jitter to reduce the impact of in-app events bursts sent by the tracking platforms

• Increased Data Retention Period from 1 day (default) to 3 days on the raw data streams

Jedi Trial II

• Firehose real time data-ingestion to S3and auto scaling capabilities flushes the data to S3/EMR cluster faster than ever letting our machine learning platform re-calculate user retargeting segments with higher frequency

• Encryption is a key success factor since we manage sensitive data contained on the in-app events

Jedi Trial III

• EMR Cluster simplifies our data processing

• Spark ETLs are executed by Airflow, to enrich data, de-normalize and convert JSON to Parquet.

• ML predicts user conversion and separates users based on it. This process is implemented as a Python app that queries event data stored in Parquet files through PrestoDB

Jedi Trial IV

• Airpal queries PrestoDB and simplifies access to data for non technical people

• Jupyter notebooks are used as templates to build frequently used queries and automate common analysis tasks

• Spark Streaming for real-time anomaly detection and fraud prevention

• Multiple Clusters (according to SLAs)

Jedi Trial V

Jedi Knighting (from Padawan to Jedi Knight)

• Time is money

• Shards Read/Write limits... test your data volume first!

• Shard-based provisioning throughput let you scale on demand

• Exponential backoff + Jitter

• Batch and compress will save you tons of headaches and money

• Extended Data Retention pays off

• Kinesis helps you make the data pipeline much more reliable

• Kinesis + Lambda + Dynamo + EMR = <3

Thanks!

geeks.jampp.com

Please remember to rate this session under My Agenda on

awssummit.london

http://blogs.aws.amazon.com/bigdata/

Thank You

Appendix

Scenarios Accelerated Ingest-Transform-Load

Continual Metrics Generation

Responsive Data Analysis

Data Types IT logs, applications logs, social media / clickstreams, sensor or device data, market data

Ad/ Marketing Tech

Publisher, bidder data aggregation

Advertising metrics like coverage, yield, conversion

Analytics on userengagement with ads, optimized bid / buy engines

IoT Sensor, device telemetry data ingestion

IT operational metrics dashboards

Sensor operational intelligence, alerts, and notifications

Gaming Online customer engagement data aggregation

Consumer engagement metrics for level success, transition rates, CTR

Clickstream analytics, leaderboard generation,player-skill match engines

ConsumerEngagement

Online customer engagement data aggregation

Consumer engagement metrics like page views, CTR

Clickstream analytics, recommendation engines

Streaming data scenarios across segments1 2 3