big data in the cloud: architecting a better platform
Post on 05-Aug-2015
Embed Size (px)
1. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Welcome! Big Data in The Cloud: Architecting a Better Platform Brian Kinlaw, Principal Solution Architect, CSC 2. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Todays Presenters Brian Kinlaw Principal Solution Architect CSC Emerging Business Group Leads the initiation, development and execution of Big Data, Analytics, Social Media, Mobile, Cloud, Cyber Security, and Internet of Things (IoT) solutions for the Office of the CTO. 3. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Agenda I. CSC BDPaaS Overview II. CSC Approach III. BDPaaS Architecture IV. BDPaaS Security V. Questions & Answers 4. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Rapidly Evolving Analytics Landscape BIG DATA 1.0 (EDW/BI) BIG DATA 3.0 (OPEN SOURCE / NEXT GEN) KEY CHARACTERISTICS Relatively Small, Structured Data Sets Proprietary RDBMS Internally Sourced / Small Teams Reactive Reporting Mechanisms Introduction of Unstructured Data Sources New In-Memory Analytic Capabilities Data Scientists Emerge Ad-hoc Reporting Becoming Pervasive Seamless Blend of Traditional Analytics and Big Data Heavily Open Sourced Reporting Becomes Predictive & Influence Business Process Change REPRESENTATIVE TECHNOLOGIES IBM DB2, Oracle DB, IBM Cognos, SAP Business Objects, Oracle BI, Informatica IBM Netezza, HP Vertica, Oracle Exadata & Exalytics, Teradata, Pivotal Greenplum Cloudera Hadoop, Hortonworks Hadoop, Spark, Storm, Kafka, Tableau, Pentaho POTENTIAL BUSINESS ROI Low-Medium Medium Very High CUSTOMER SKILLS/TALENT Bulk of Talent Today Talent Investments Required High Demand Talent BIG DATA 2.0 (ANALYTIC APPLIANCES) DETERMINING VALUE SECURITY & COMPLIANCESKILLS & CAPABILITIES 32%30%65% The Market is Here Today Yet Challenges Remain 5. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 CSC BIG DATA & ANALYTICS: WE ARE UNIQUELY POSITIONED TO ADD VALUE Technology Expertise Working with Hadoop since its Creation Faster Time to Value Deliver a Big Data Platform in 30 Days Enterprise Security Data, Application, Platform Security and Compliance SHAPE TRANSFORM MANAGEMENT AS A SERVICE DIFFERENTIATION: OUR UNIQUE STRENGTHS FIVECORE OFFERINGS Analytics aaSBig Data Analytic Insights Big Data Strategy Big Data Platform Innovation Big Data Platform aaS STRATEGY ANALYTICS PLATFORMS INDUSTRY ACCELERATOR S Product Innovation: Optimize product mix & feature set to improve revenue by 25-30% Customer Intelligence: Identify innovative new revenue channels up to 2x revenue increase Smart Operations: Improve operating margins ~60% thru efficiency and quality improvements Risk Insights: Reduce fraudulent activity by up to 75%, avoid millions in cost & exposure Revenue Enhancers Profit Enhancers 6. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Client Value Achieved Prioritized Roadmap of Initiatives to Achieve Growth Vision within 2-3 years: BU Growth from $200M to $1B Through Analytic Insights Client Value Achieved 331% ROI Payback Period of 2.1 Months 2% Yield Improvement = $300M Client Value Achieved Reduced time to onboard customers by 80% Improved visibility on service levels Increased customer satisfaction Client Value Achieved BSL Met Strategic Objective (ITaaS) Reduced Costs by 20% Improved Analytic Cycle Time by 50% Client Value Achieved Access to Information in Minutes versus Weeks Speed: Solution Deployed within Days Access to Key Next Gen Talent Client Value Achieved Speed to Market: 30 Days to Platform, 60 Days to Full Working Mobile Telematics Application Flexible Deployment Options Achieving Real Business Value With Our Clients Integrated data for ~100M people from 40 member companies Healthcare Maximized diamond company profitability through BI and analytics Wholesale Railway punctuality improved from 92% to a world-leading 96% Transportation Reduced tax evasion and litigation through DW and predictive modeling Government 16% increase in claims fraud investigations for significant ROI in 6 months Insurance Performance optimization and analytical insights into POS and sales trends Retail/CPG $10M reduction in annual operating expenses Printing Customer intelligence lifetime value model driving marketing and customer service Travel & Leisure Use of sensor data for real- time management of mining and mfg. ops and maintenance Natural Resources Comprehensive global view of exposure in near real time Banking Global Insurance Company 7. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 RISK RESULT Structuring all data at the point of ingestion Schema on Write vs Schema on Read Significant upfront expense ( and $$) for planning Significant expense ( and $$) to adapt to changes/needs of the business Data silos Disparate information streams Reduced ability to obtain requirements from entire business Does not allow for holistic decisions to be made No golden source of truth Proprietary/custom data warehousing/infrastructure Expensive Non standard to environment Scale Not economically feasible Not technically possible Risk to Traditional Data Model the status quo 8. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Risk of Transforming to a Big Data Business RISK RESULT Numerous different technologies Hard to select the best tool without specific experience with these technologies Lack of Big Data specific expertise Unreasonable expectations without having done it before R&D in Big Data is lost or as time permits Scope creep is common Learning as your go Immature Big Data Technologies Compliance risk Security Risk Complex deployments Complex integrations between technologies High operational costs Large CapEx expenditure Buying upfront growth More complex to scale Big Data & Analytic systems should be a tool to enable companies with better information and insights, not a roadblock 9. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 1. Implementation Complexity Integration Speed 2. Operation 3. Data Science Business Relevance Feedback loop 4. Talent Robust & Scalable Monitoring & Automated Alerts Operational Big Data Risks The right talent at the right time 5. Infrastructure Upfront - CapEx investment Iterative Flexibility Matching Hardware to Software 10. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 A New Mitigation Strategy Big Data Platform-as-a-Service Operation Managed to your SLA needs Global delivery teams and support Integrated testing Implementation DevOps infrastructure-as-code deployment Pre-defined orchestration scripts Flexible deployment locations Talent Data engineers Solution Architects ETL expertise Support Team R&D Team BI/Viz/Reporting expertise Data Science Subject matter expertise as needed Global Data Science team Applying analysis at the right point Infrastructure as-a-Service model Pay-as-you-go structure Pre-configured hardware designs 11. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Agenda I. CSC BDPaaS Overview II. CSC Approach III. BDPaaS Architecture IV. BDPaaS Security V. Questions & Answers 12. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Descriptive Analytics I What happened? Reporting Query, Reporting, and Search Tools Diagnostic Analytics Why did it happen? Analysis OLAP and Visualization Tools Descriptive Analytics II Whats happening now? Monitoring Dashboards and Scorecards Predictive Analytics What might happen? Predictive Analysis Big Data Prescriptive Analytics How can we make it happen? Recommendations, Risk Avoidance Complexity BusinessValue Operations Triggers High ImpactLow Impact Process Improvement via Applied Intelligence The Analytics Journey 13. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 MAJOR ACTIVITIES Solution Iterative App DevelopmentPlatform RolloutTechnical DesignDiscovery Interview Key Business Stakeholders Interview Key Technical Stakeholders Define Objectives & Challenges Define Target Use Case Identify Data Sources Define Business Benefits Define Architecture Develop High- Level Approach & Costs Agree to Project Plan/Rollout Standup / Connect Environment Design Data Flows Architecture Validation Build Data Flows Historical Data Real-Time Data Flow MANAGETRANSFORMSHAPE Iterate Identify data sources for target use case Develop high level tech approach and costs Define high level benefits Develop initial case for action Develop go forward plan Develop Data Model Technical architecture & integration design Stand up environment Dashboard design workshops Data mapping Build dashboard Configure application Data load Run solution iterations Analytical modeling 2-4 hour Design Thinking Workshop Review current state metrics Review business pain points & opportunities Review application & infrastructure environment Define target use case Customer Engagement Framework 14. AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015 Data Exploration & Transformation Data Modeling & Algorithm Development Data Visualization & Reporting Business Discovery InsightLab: Rapid Analytics Development Insight Operationalization Change Management Use Case Prioritization & Roadmap Data Inventory Identification & Coordination 8 12 Week Sprint Agile Scientific Approach to Measurable Business Improvement Inputs Outputs InsightLab 15.