big data and business insight

Download Big Data and Business Insight

Post on 13-Jan-2017




2 download

Embed Size (px)


  • 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

    November 8th, 2016

    Big Data & Business Insight

    Ian Meyers - Senior Manager, Solutions Architecture, AWS John Kundert - CTO, The Financial Times

  • Why Cloud for Big Data & Business Insight?

  • Modern Analytics ApplicationsVariety, volume, and velocity

    requiring new tools

    New analysis requirements

    Cloud ComputingVariety of compute, storage, and

    networking options

  • Modern Analytics ApplicationsPotentially massive datasets

    Massive datasets

    Cloud ComputingMassive, virtually unlimited capacity

  • Modern Analytics ApplicationsIterative, experimental style of data

    manipulation and analysis

    Need for greater agility

    Cloud ComputingIterative, experimental style of

    infrastructure deployment/usage

  • Modern Analytics ApplicationsFrequently not steady-state workload; peaks and valleys

    Variable workloads and volume

    Cloud ComputingAt its most efficient with highly

    variable workloads

  • Modern Analytics ApplicationsAbsolute performance not as critical

    as time to results. Shared resources are a bottleneck

    Projects require fast time to results

    Cloud ComputingParallel compute projects allow each workgroup to have more autonomy, get faster results

  • One tool to rule them all

  • Use the right tools for you

    Virtually unlimited

    Flexible, strong security

    Your data format & structure

    Powerful and fast

    Selection of tools

    Pay for what you use

    Rapid data discovery Pixel perfect

    reporting Third party & AWS


    Storage Analysis Presentation

  • Use the right tools for you

    Block & Filesystem Storage

    Managed RDBMS Managed NOSQL

    Managed data warehouse Data streams

    SQL for analysis & transaction processing

    Managed Hadoop/Spark Predictive analytics & Deep

    Learning Data Workflow

    Rapid development &

    data discovery Kibana dashboards

    Industry leading third

    party solutions

    Storage Analysis Presentation


    Amazon Glacier

    Amazon Elastic



    Amazon Redshift

    Amazon EMR

    Amazon Kinesis

    Amazon QuickSightAmazon

    Elasticsearch Service


    DynamoDB Amazon Machine Learning

  • Online Software

  • Media streaming

    Marketing campaigns

    Disaster recovery

    Web site & media sharing

    Facebook app

    Ground campaign

    SAP & SharePoint

    Marketing web site

    Social Media Monitoring

    Consumer social app

    IT operations

    Mars exploration ops

    Interactive TV apps

    Consumer social app

    Facebook presence

    Securities Trading Data Archiving

    Financial markets analytics

    Web and mobile apps

    Big data analytics

    Digital media

    Ticket pricing optimization

    Streaming webcasts

    Mobile analytics

    Consumer social app

    Core IT and media

  • Certifications and accreditations for workloads that matter

    AWS CloudTrail - AWS API call logging for governance & compliance

    Log and review user activity

    Store data in S3, archive to Glacier, or stream process with AWS Lambda

    AWS Security

    AWS CloudTrail

  • Highlighted Customer Stories Regulatory Agencies

    FINRA, the primary regulatory agency for broker-dealers in the US, uses AWS extensively in their IT operations and has migrated key portions of its technology stack to AWS including Market Surveillance and Member Regulation. For market surveillance, each night FINRA loads approximately 35 billion rows of data into Amazon S3 and Amazon EMR to monitor trading activity on exchanges and market centers in the US.

    In response to the May 6, 2010 Flash Crash in U.S. markets, the SEC used Tradeworx and AWS to create its Market Information Data Analytics System (MIDAS), which enables the agency to collect and analyze billions of rows of data and to reconstruct any market event down to the individual record, analyzing more than 3 billion data points in seconds rather than weeks or months.

    For our market surveillance systems, we are looking at about 40% [savings with AWS], but the real benefits are the business benefits: We can do things that we physically werent able to do before, and that is priceless. Steve Randich, CIO

  • Japans largest mobile service provider

    125 node Amazon Redshift DS2.8XL cluster 4,500 vCPUs, 30 TB RAM 2 PB compressed 10x faster analytic queries 50% reduction in time for new BI application deployment Significantly lower operations overhead

    68 million customers Tens of TBs per day of data across a mobile network 6 PB of total data (uncompressed) Data science for marketing operations, logistics, and so on Scaling challenges Performance issues

    The Challenge The Solution

  • A case study of a transformative digital business model @ John Kundert, CTO

  • Financial Times Profile

    Age: 128 years

    Size: ~300 million GBP / ~2000 employees

    Location: Global operations with UK at the centre

    Challenge: Transformation from print to digital

    Strategy: Grow paid for content business models

  • Challenges transforming into a digital business

    Putting the customer at the heart of the organization

    Building a world class Product team

    Building a world class Engineering team

    Getting Data into the heart of the organization

    Embedding change into the culture - move fast, be less risk adverse, embrace failure

  • Open for editorial

  • Open for customers

  • Open for knowledge managers

  • Adding a little colour - factoids

    Production system 30 TB

    Speedy Analytics 3 TB 91k queries day

    700M records per day

    520+ users

    best case 1 second latency

  • Our Data Story Certainty

  • The beautifully reassuring illusion of control

    We consulted with the whole business We defined a business case We executed an RFP We asked for a bit more money We secured our preferred partner(s) We started legal contracts and formed a team We performed SSA (source system analysis) We

    Senior management striving for certainty

  • No escape velocity

    The world around us changed faster than our definition of the required change

  • Our Data Story Empowerment

    or Trust - letting go (a little)

  • 1. We re-organised ourselves

  • 2. We formed a vision statement

    enable all of FT and its products to

    easily discover and trust our data and business intelligence in near real time

  • 3. We defined measureable outcomes

    a. ownership of IP

    b. reduction in cost (TCO)

    c. reach

    d. return

  • 4. We built a small diverse team familiarity with the business including our heritage and culture

    technical diversity;

    traditional data warehousing

    software/front end developers

    analytical expertise

    tech lead with light touch support functions

  • 5. We empowered the team - ownership

    owned the long term vision

    owned the technology and the platform

    owned the business relationships

    managed cost constraints

    shared success and kudos

  • 6. Executive accepted short term (low cost) risk

    Team empowered to achieve outcomes through their choices

    start fast fail/succeed learn from results share

  • 7. Communication based on quality not quantity


    Full transparency at all times

    Pull and Push

    Ask for help when needed

    Short and clean

    Open Failure treated as success

  • The Unguided Missiles

  • Data Governance and QualityNo agreed definition for;


    engaged customer

    active subscriber


    Resolution does not lend itself to agile methodologies.

    DQ image: Sourced from information management group

  • Cross-programme interdependence

    Dependencies outside the data programme of work to build;

    service levels

    new integration points

    migration from batch to APIs

    change of source

  • Outcomes

  • Moving costs from engineering to business value

  • Reach / Discovery o FT boardroom

    o FT newsroom

    o B2B and B2C business

    o Advertising

    o Analytics

    o Product development

    o Technology

  • Return / Value

    Informed changes in the editorial workflows - moving towards a digital first production process

    Calibrated the B2B paid for content business model - free vs paid

    Predictive analytics - leading indicators driving B2C subscriptions

    Optimizing the subscription models - metered model to trial model

    Product Development - moving from hunches to following the data

    Cultural change - evidence based accelerating our ability to change - test more / talk less

  • Governance / Trust

    o universal definition with data governance for all core metrics

    o enterprise wide adoption of new metrics for Engagement

    o subscription wide adoption of new lifetime value metrics

    o product and investment decisions based on an agreed version of the truth

  • http://aws.amaz