Download - Big Data and Business Insight
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
November 8th, 2016
Big Data & Business Insight
Ian Meyers - Senior Manager, Solutions Architecture, AWS John Kundert - CTO, The Financial Times
Why Cloud for Big Data & Business Insight?
Modern Analytics ApplicationsVariety, volume, and velocity
requiring new tools
New analysis requirements
Cloud ComputingVariety of compute, storage, and
networking options
Modern Analytics ApplicationsPotentially massive datasets
Massive datasets
Cloud ComputingMassive, virtually unlimited capacity
Modern Analytics ApplicationsIterative, experimental style of data
manipulation and analysis
Need for greater agility
Cloud ComputingIterative, experimental style of
infrastructure deployment/usage
Modern Analytics ApplicationsFrequently not steady-state workload; peaks and valleys
Variable workloads and volume
Cloud ComputingAt its most efficient with highly
variable workloads
Modern Analytics ApplicationsAbsolute performance not as critical
as “time to results”. Shared resources are a bottleneck
Projects require fast time to results
Cloud ComputingParallel compute projects allow each workgroup to have more autonomy, get faster results
One tool to rule them all
Use the right tools for you
Virtually unlimited
Flexible, strong security
Your data format & structure
Powerful and fast
Selection of tools
Pay for what you use
Rapid data discovery Pixel perfect
reporting Third party & AWS
Tools
Storage Analysis Presentation
Use the right tools for you
Block & Filesystem Storage
Managed RDBMS Managed NOSQL
Managed data warehouse Data streams
SQL for analysis & transaction processing
Managed Hadoop/Spark Predictive analytics & Deep
Learning Data Workflow
Rapid development &
data discovery Kibana dashboards
Industry leading third
party solutions
Storage Analysis Presentation
AmazonS3
Amazon Glacier
Amazon Elastic
Filesystem
AmazonRDS
Amazon Redshift
Amazon EMR
Amazon Kinesis
Amazon QuickSightAmazon
Elasticsearch Service
AmazonDynamoDBAmazon
DynamoDB Amazon Machine Learning
Online Software Storeaws.amazon.com/marketplace
Media streaming
Marketing campaigns
Disaster recovery
Web site & media sharing
Facebook app
Ground campaign
SAP & SharePoint
Marketing web site
Social Media Monitoring
Consumer social app
IT operations
Mars exploration ops
Interactive TV apps
Consumer social app
Facebook presence
Securities Trading Data Archiving
Financial markets analytics
Web and mobile apps
Big data analytics
Digital media
Ticket pricing optimization
Streaming webcasts
Mobile analytics
Consumer social app
Core IT and media
Certifications and accreditations for workloads that matter
AWS CloudTrail - AWS API call logging for governance & compliance
Log and review user activity
Store data in S3, archive to Glacier, or stream process with AWS Lambda
AWS Security
AWS CloudTrail
Highlighted Customer Stories – Regulatory Agencies
FINRA, the primary regulatory agency for broker-dealers in the US, uses AWS extensively in their IT operations and has migrated key portions of its technology stack to AWS including Market Surveillance and Member Regulation. For market surveillance, each night FINRA loads approximately 35 billion rows of data into Amazon S3 and Amazon EMR to monitor trading activity on exchanges and market centers in the US.
In response to the May 6, 2010 Flash Crash in U.S. markets, the SEC used Tradeworx and AWS to create its Market Information Data Analytics System (MIDAS), which enables the agency to collect and analyze billions of rows of data and to reconstruct any market event down to the individual record, analyzing more than 3 billion data points in seconds rather than weeks or months.
For our market surveillance systems, we are looking at about 40% [savings with AWS], but the real benefits are the business benefits: We can do things that we physically weren’t able to do before, and that is priceless.” – Steve Randich, CIO
https://aws.amazon.com/solutions/case-studies/big-data
Japan’s largest mobile service provider
125 node Amazon Redshift DS2.8XL cluster 4,500 vCPUs, 30 TB RAM 2 PB compressed 10x faster analytic queries 50% reduction in time for new BI application deployment Significantly lower operations overhead
68 million customers Tens of TBs per day of data across a mobile network 6 PB of total data (uncompressed) Data science for marketing operations, logistics, and so on Scaling challenges Performance issues
The Challenge The Solution
https://aws.amazon.com/solutions/case-studies/big-data
A case study of a transformative digital business model @ John Kundert, CTO
Financial Times Profile
❏ Age: 128 years
❏ Size: ~300 million GBP / ~2000 employees
❏ Location: Global operations with UK at the centre
❏ Challenge: Transformation from print to digital
❏ Strategy: Grow paid for content business models
Challenges transforming into a digital business
❏ Putting the customer at the heart of the organization
❏ Building a world class Product team
❏ Building a world class Engineering team
❏ Getting Data into the heart of the organization
❏ Embedding change into the culture - move fast, be less risk adverse, embrace failure
Open for editorial
Open for customers
Open for knowledge managers
Adding a little colour - factoids
Production system 30 TB
Speedy Analytics 3 TB 91k queries day
700M records per day
520+ users
best case 1 second latency
Our Data Story Certainty
The beautifully reassuring illusion of control
• We consulted with the whole business • We defined a business case • We executed an RFP • We asked for a bit more money • We secured our preferred partner(s) • We started legal contracts and formed a team • We performed SSA (source system analysis) • We …
Senior management striving for certainty …
No escape velocity
The world around us changed faster than our definition of the ‘required change’
Our Data Story Empowerment
or Trust - letting go (a little)
1. We re-organised ourselves
2. We formed a vision statement
‘enable all of FT and its products to
easily discover and trust our data and
business intelligence in near real time’
3. We defined measureable outcomes
a. ownership of IP
b. reduction in cost (TCO)
c. reach
d. return
4. We built a small diverse team❏ familiarity with the business including our heritage and culture
❏ technical diversity;
❏ traditional data warehousing
❏ software/front end developers
❏ analytical expertise
❏ tech lead with light touch support functions
5. We empowered the team - ownership
❏ owned the long term vision
❏ owned the technology and the platform
❏ owned the business relationships
❏ managed cost constraints
❏ shared success and kudos
6. Executive accepted short term (low cost) risk
Team empowered to achieve outcomes through their choices
start fast fail/succeed learn from results share
7. Communication based on quality not quantity
Multi-channel
Full transparency at all times
Pull and Push
Ask for help when needed
Short and clean
Open Failure treated as success
The Unguided Missiles
Data Governance and QualityNo agreed definition for;
❏ customer
❏ engaged customer
❏ active subscriber
❏ ...
Resolution does not lend itself to agile methodologies.
DQ image: Sourced from information management group
Cross-programme interdependence
Dependencies outside the data programme of work to build;
❏ service levels
❏ new integration points
❏ migration from batch to API’s
❏ change of source
Outcomes
Moving costs from engineering to business value
Reach / Discovery o FT boardroom
o FT newsroom
o B2B and B2C business
o Advertising
o Analytics
o Product development
o Technology
Return / Value
• Informed changes in the editorial workflows - moving towards a digital first production process
• Calibrated the B2B paid for content business model - free vs paid
• Predictive analytics - leading indicators driving B2C subscriptions
• Optimizing the subscription models - metered model to trial model
• Product Development - moving from hunches to following the data
• Cultural change - evidence based accelerating our ability to change - test more / talk less
Governance / Trust
o universal definition with data governance for all core metrics
o enterprise wide adoption of new metrics for Engagement
o subscription wide adoption of new lifetime value metrics
o product and investment decisions based on an agreed version of the truth
http://aws.amazon.com/marketplace
Learn from other AWS customers
aws.amazon.com/solutions/case-studies/big-data
Big Data Case Studies
APN Partner-provided labsaws.amazon.com/testdrive/bigdata
AWS Big Data Test Drives
https://aws.amazon.com/training
Webinars, Bootcamps, and Self-Paced Labsaws.amazon.com/events
New course on Big Dataaws.amazon.com/training/course-descriptions/bigdata
AWS Training & Events
Thank You!Ian Meyers - Senior Manager, Solutions Architecture, AWS
John Kundert - CTO, The Financial Times