big data and business insight

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

November 8th, 2016

Big Data & Business Insight

Ian Meyers - Senior Manager, Solutions Architecture, AWS John Kundert - CTO, The Financial Times

Why Cloud for Big Data & Business Insight?

Modern Analytics ApplicationsVariety, volume, and velocity

requiring new tools

New analysis requirements

Cloud ComputingVariety of compute, storage, and

networking options

Modern Analytics ApplicationsPotentially massive datasets

Massive datasets

Cloud ComputingMassive, virtually unlimited capacity

Modern Analytics ApplicationsIterative, experimental style of data

manipulation and analysis

Need for greater agility

Cloud ComputingIterative, experimental style of

infrastructure deployment/usage

Modern Analytics ApplicationsFrequently not steady-state workload; peaks and valleys

Variable workloads and volume

Cloud ComputingAt its most efficient with highly

variable workloads

Modern Analytics ApplicationsAbsolute performance not as critical

as “time to results”. Shared resources are a bottleneck

Projects require fast time to results

Cloud ComputingParallel compute projects allow each workgroup to have more autonomy, get faster results

One tool to rule them all

Use the right tools for you

Virtually unlimited

Flexible, strong security

Your data format & structure

Powerful and fast

Selection of tools

Pay for what you use

Rapid data discovery Pixel perfect

reporting Third party & AWS

Tools

Storage Analysis Presentation

Use the right tools for you

Block & Filesystem Storage

Managed RDBMS Managed NOSQL

Managed data warehouse Data streams

SQL for analysis & transaction processing

Managed Hadoop/Spark Predictive analytics & Deep

Learning Data Workflow

Rapid development &

data discovery Kibana dashboards

Industry leading third

party solutions

Storage Analysis Presentation

AmazonS3

Amazon Glacier

Amazon Elastic

Filesystem

AmazonRDS

Amazon Redshift

Amazon EMR

Amazon Kinesis

Amazon QuickSightAmazon

Elasticsearch Service

AmazonDynamoDBAmazon

DynamoDB Amazon Machine Learning

Online Software Storeaws.amazon.com/marketplace

Media streaming

Marketing campaigns

Disaster recovery

Web site & media sharing

Facebook app

Ground campaign

SAP & SharePoint

Marketing web site

Social Media Monitoring

Consumer social app

IT operations

Mars exploration ops

Interactive TV apps

Consumer social app

Facebook presence

Securities Trading Data Archiving

Financial markets analytics

Web and mobile apps

Big data analytics

Digital media

Ticket pricing optimization

Streaming webcasts

Mobile analytics

Consumer social app

Core IT and media

Certifications and accreditations for workloads that matter

AWS CloudTrail - AWS API call logging for governance & compliance

Log and review user activity

Store data in S3, archive to Glacier, or stream process with AWS Lambda

AWS Security

AWS CloudTrail

Highlighted Customer Stories – Regulatory Agencies

FINRA, the primary regulatory agency for broker-dealers in the US, uses AWS extensively in their IT operations and has migrated key portions of its technology stack to AWS including Market Surveillance and Member Regulation. For market surveillance, each night FINRA loads approximately 35 billion rows of data into Amazon S3 and Amazon EMR to monitor trading activity on exchanges and market centers in the US.

In response to the May 6, 2010 Flash Crash in U.S. markets, the SEC used Tradeworx  and AWS to create its Market Information Data Analytics System (MIDAS), which enables the agency to collect and analyze billions of rows of data and to reconstruct any market event down to the individual record, analyzing more than 3 billion data points in seconds rather than weeks or months.

For our market surveillance systems, we are looking at about 40% [savings with AWS], but the real benefits are the business benefits: We can do things that we physically weren’t able to do before, and that is priceless.” – Steve Randich, CIO

https://aws.amazon.com/solutions/case-studies/big-data

Japan’s largest mobile service provider

125 node Amazon Redshift DS2.8XL cluster 4,500 vCPUs, 30 TB RAM 2 PB compressed 10x faster analytic queries 50% reduction in time for new BI application deployment Significantly lower operations overhead

68 million customers Tens of TBs per day of data across a mobile network 6 PB of total data (uncompressed) Data science for marketing operations, logistics, and so on Scaling challenges Performance issues

The Challenge The Solution

https://aws.amazon.com/solutions/case-studies/big-data

A case study of a transformative digital business model @ John Kundert, CTO

Financial Times Profile

❏ Age: 128 years

❏ Size: ~300 million GBP / ~2000 employees

❏ Location: Global operations with UK at the centre

❏ Challenge: Transformation from print to digital

❏ Strategy: Grow paid for content business models

Challenges transforming into a digital business

❏ Putting the customer at the heart of the organization

❏ Building a world class Product team

❏ Building a world class Engineering team

❏ Getting Data into the heart of the organization

❏ Embedding change into the culture - move fast, be less risk adverse, embrace failure

Open for editorial

Open for customers

Open for knowledge managers

Adding a little colour - factoids

Production system 30 TB

Speedy Analytics 3 TB 91k queries day

700M records per day

520+ users

best case 1 second latency

Our Data Story Certainty

The beautifully reassuring illusion of control

• We consulted with the whole business • We defined a business case • We executed an RFP • We asked for a bit more money • We secured our preferred partner(s) • We started legal contracts and formed a team • We performed SSA (source system analysis) • We …

Senior management striving for certainty …

No escape velocity

The world around us changed faster than our definition of the ‘required change’

Our Data Story Empowerment

or Trust - letting go (a little)

1. We re-organised ourselves

2. We formed a vision statement

‘enable all of FT and its products to

easily discover and trust our data and

business intelligence in near real time’

3. We defined measureable outcomes

a. ownership of IP

b. reduction in cost (TCO)

c. reach

d. return

4. We built a small diverse team❏ familiarity with the business including our heritage and culture

❏ technical diversity;

❏ traditional data warehousing

❏ software/front end developers

❏ analytical expertise

❏ tech lead with light touch support functions

5. We empowered the team - ownership

❏ owned the long term vision

❏ owned the technology and the platform

❏ owned the business relationships

❏ managed cost constraints

❏ shared success and kudos

6. Executive accepted short term (low cost) risk

Team empowered to achieve outcomes through their choices

start fast fail/succeed learn from results share

7. Communication based on quality not quantity

Multi-channel

Full transparency at all times

Pull and Push

Ask for help when needed

Short and clean

Open Failure treated as success

The Unguided Missiles

Data Governance and QualityNo agreed definition for;

❏ customer

❏ engaged customer

❏ active subscriber

❏ ...

Resolution does not lend itself to agile methodologies.

DQ image: Sourced from information management group

Cross-programme interdependence

Dependencies outside the data programme of work to build;

❏ service levels

❏ new integration points

❏ migration from batch to API’s

❏ change of source

Outcomes

Moving costs from engineering to business value

Reach / Discovery o FT boardroom

o FT newsroom

o B2B and B2C business

o Advertising

o Analytics

o Product development

o Technology

Return / Value

• Informed changes in the editorial workflows - moving towards a digital first production process

• Calibrated the B2B paid for content business model - free vs paid

• Predictive analytics - leading indicators driving B2C subscriptions

• Optimizing the subscription models - metered model to trial model

• Product Development - moving from hunches to following the data

• Cultural change - evidence based accelerating our ability to change - test more / talk less

Governance / Trust

o universal definition with data governance for all core metrics

o enterprise wide adoption of new metrics for Engagement

o subscription wide adoption of new lifetime value metrics

o product and investment decisions based on an agreed version of the truth

http://aws.amazon.com/marketplace

Learn from other AWS customers

aws.amazon.com/solutions/case-studies/big-data

Big Data Case Studies

APN Partner-provided labsaws.amazon.com/testdrive/bigdata

AWS Big Data Test Drives

https://aws.amazon.com/training

Webinars, Bootcamps, and Self-Paced Labsaws.amazon.com/events

New course on Big Dataaws.amazon.com/training/course-descriptions/bigdata

AWS Training & Events

Thank You!Ian Meyers - Senior Manager, Solutions Architecture, AWS

John Kundert - CTO, The Financial Times