![Page 1: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/1.jpg)
Making Big Data Work for Your Organization
Alan Simon Zunesis Senior Fellow, Analytics & Data Management December 3, 2015
© 2015 Alan Simon. All Rights Reserved
![Page 2: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/2.jpg)
Introductions
2
![Page 3: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/3.jpg)
Select experience • Arizona Attorney General’s Office • Pennsylvania Department of Transportation (PennDOT) • Pennsylvania Department of Health • Wisconsin Department of Administration • Santa Clara County (CA) • Washington (State) Department of Early Learning • Arizona State University • USAF • Department of Defense
3
![Page 4: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/4.jpg)
Check out this mission statement…
• “Our investigators need instantaneous access to a broad range of data at their fingertips…not only all direct person-to-person and person-to-business relationships, but also relationships several levels removed. The same direct and indirect relationships to assets also need to be immediately available to our investigators…”
4
![Page 5: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/5.jpg)
5
![Page 6: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/6.jpg)
Nope! • Arizona Attorney General’s Office, 1979-1980
6
![Page 7: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/7.jpg)
What do we want from analytics and big data?
Data-Driven Insights
Insight-Driven “Better” Decisions
7
![Page 8: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/8.jpg)
Proposition:
8
![Page 9: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/9.jpg)
Big data success isn’t “automatic”!!! 9
However…
![Page 10: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/10.jpg)
7 KEYS TO BIG DATA SUCCESS
10
![Page 11: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/11.jpg)
“The art of the now possible” • Put deferred solutions back on the table
and also
• Reengineer and rebuild old, overly complicated solutions
11
1
![Page 12: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/12.jpg)
2001-2002: The Anthrax Attacks
12
![Page 13: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/13.jpg)
Bioterrorism fears go viral !
• Anthrax • Smallpox • Plague • Viral hemorrhagic fever • …?????
13
![Page 14: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/14.jpg)
Worries about timeliness of response • Many bioterrorism agents show symptoms…but cannot be
confirmed for several more days • In meantime outbreaks could get much worse • The “solution” – a concept known as
Syndromic Surveillance
14
Interdiction
Patterns
Reports
Hypotheses
Analysis
![Page 15: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/15.jpg)
Circa 2001-2002 solution:
15
![Page 16: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/16.jpg)
But if I could figure out time travel…
16
![Page 17: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/17.jpg)
“Big data” doesn’t just mean Hadoop!
• Hadoop is certainly important but also… • HANA • MongoDB • Columnar and other specialized databases • Pretty much any “post-relational” data management technology!
• Also: “big” data is relative! 17
2
![Page 18: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/18.jpg)
Hadoop is evolving at light speed!
18
3
https://blog.cloudera.com/blog/2015/09/kudu-new-apache-hadoop-storage-for-fast-analytics-on-fast-data/ http://www.infoworld.com/article/2986675/hadoop/cloudera-kudu-hdfs-hbase-in-one.html
![Page 19: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/19.jpg)
Evolution often yields confusion…
19
http://sdsblog.com/2015/10/12/kudu/
![Page 20: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/20.jpg)
Traditional BI and DW not out of the picture…at least for now…
20
4
![Page 21: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/21.jpg)
Hadoop data staging area AND ANALYTICS SANDBOX
Relational
EDW
Sqoop Flume …
Sqoop
Sqoop Flume …
Hadoop as “supersized data staging area” in front of EDW
![Page 22: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/22.jpg)
Hadoop EDW: staging + user-accessible data
Hadoop/data lake as next-generation EDW
Sqoop Flume …
Sqoop Flume …
![Page 23: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/23.jpg)
Prescriptive analytics are critically important!
23
5
![Page 24: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/24.jpg)
Category Purpose
Descriptive analytics
Tell me what happened, and why Tell me what is happening right now, and why
Predictive analytics
Tell me what is likely to happen, and why
Discovery analytics
Tell me something important…even without me asking specific questions!
Prescriptive analytics
Tell me what my options are Tell me what I should do
The Analytics Continuum
![Page 25: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/25.jpg)
The prescriptive analytics framework Detect Events
Categorize and Process Events
Apply Analytical Models
Form Hypotheses
Take Initial Actions
Update and Correlate Data
Prove or Disprove Hypotheses
Take Prescribed Actions
Analytics BPM +
25
![Page 26: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/26.jpg)
Why?
26
![Page 27: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/27.jpg)
The hard work doesn’t go away with big data
• You still need: • Master data management
• Data standardization
• Data governance
• Data quality management
• …
27
6
![Page 28: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/28.jpg)
Your big data strategy and architecture must also include your social media listening and engagement
28
7
![Page 29: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/29.jpg)
Everyone is Connected! • Constituents • Service Consumers • Residents • Voters • Veterans
This is the world we now live in… Your Citizens are Online
![Page 30: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/30.jpg)
Communications Manager Public Info. Office Public Services
iWeSocial IQ Social Media Listening Platform iWeSocial
Research Analyst
Public Web:
iWeSocial Insights (monthly readout)
Government Organizations
Focus: Services, Security, Sentiment
Social Listening for State and Local Government Technology + Human Analysis = Actionable Insight
![Page 31: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/31.jpg)
Social Listening Use Case: McCarran International Airport
Business Return: • Discover what’s
being said about the services you provide: ü Sentiment ü Problem areas ü What's
working
• Crisis Management – listen for spikes in conversation in real-time
• Increase engagement by better understanding the conversation
![Page 32: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/32.jpg)
HOW DO YOU GET STARTED?
32
![Page 33: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/33.jpg)
Good news: the classic approach still works!
33
Assessment
Strategy
Business Architecture
Technology Architecture
Roadmap
![Page 34: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/34.jpg)
During assessment, focus on:
• Both opportunities and “points of pain” • What you cannot do today • What is challenging and cumbersome to do today • What has been “put into mothballs” • “Blue skies and green fields”
34
![Page 35: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/35.jpg)
Big data and analytics assessment
35
Assessment • Analytics continuum • Today’s technology • In-progress initiatives
• Hypotheses vs. facts • Pain points • Known opportunities
• Deferred initiatives • Analytics appetite • Competition
• Typically 3 – 5 weeks • Grounded in your organization’s reality
![Page 36: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/36.jpg)
Followed by…
36
Strategy • Commitment to analytics • Change management • Pace of change
• Migration strategy • Leadership (RACI) • Resourcing models
Business Architecture
• “Day in the Life” scenarios • Use cases and process flows • Roles and responsibilities
Technology Architecture
• DW/BI role • Big data role • Platforms
• Tools • Migration plans • Integration architecture
• Support • Data flows • Control flows
Roadmap • Risk mitigation • Program leadership • Sponsorship
• Defining success • CSFs • User adoption
• Phases • Stretch goals • Contingency plans
• Typically 8-12 weeks • Complete blueprint
![Page 37: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/37.jpg)
We’d love to hear from you!
Zunesis
Email: [email protected]
Website: www.zunesis.com
Headquarters 8375 S. Willow Street 5th Floor Lone Tree, CO 80124 720-221-5200
37
Las Vegas Office 6280 S. Valley View Blvd. Suite 604 Las Vegas, NV 89118 702-837-5300
![Page 38: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/38.jpg)
• My Wednesday analytics blog: https://humpdayanalytics.wordpress.com/
• Cross-posted on LinkedIn
• Follow me on Twitter: @HumpDayAnalytic
• My LinkedIn/Lynda.com courses:http://www.lynda.com/Alan-Simon/3981678-1.html
For more information…
38
![Page 39: Nevada DGS 2015 Presentation - Making Big Data Work -Alan Simon](https://reader034.vdocuments.net/reader034/viewer/2022051116/5695d0de1a28ab9b02943317/html5/thumbnails/39.jpg)
39