using big data to create a data drive organization
DESCRIPTION
Presentation given Jan 21, 2013 at the MinneAnalytics Conference in Minnesota.TRANSCRIPT
The Future, According to Sci-fi
Meet your Real Future
How Hadoop Works
Companies are seeing returns from big data
0%10%20%30%40%50%60%70%80%90%
ImprovedBusinessDecisions
ImprovedCurrentRevenueStreams
Support ofNew
RevenueStreams
NotLeveraged
for RevenueGrowth
Uses of Big Data
Uses of Big Data
Source: Avanade Inc. 2012 Big Data Survey
The Heart of a Data Driven Organization Data drives decisions and are the key to all decisions
made within the organization
People who think make decisions, not data!
A data driven organization can not truly use data on its own, it takes people with the right skills and expertise in knowing how to use the data, to truly be data driven.
Evidence based decisions + Reasoned Arguments is how an organization becomes data driven.
“An organization’s data is found in its computer systems, but a company’s intelligence is found its biological and social systems” --- Valdis Krebs, researcher
Obtaining Data as a competitive Advantage Best in class data driven companies take 12 days on average
to integrate new data sources into their analytical systems; industry avg companies take 60 days, laggards 143 days.
Best-in-class companies can pursue new market opportunities faster
Can take advantages quickly, newly emerging business opportunities
Can bring high-value services and products to market faster
Be proactive and create more information based insights
Source: Aberdeen Group: Data Management for BI: Fueling the analytical engine with high-octane information
To Put it Another Way Computational = Subconscious
Strategic = Conscious
How to use Big Data to create a data driven culture
Data • Data is the foundation
Insights • Insights improve understanding
Actions • Actions, create
new experiences
The data Part of the Equation
Solving Problems with Big Data Hadoop-able Problems
Complex data and lots of it
Multiple data sources and highly unstructured
Benefits of Analyzing with Hadoop
Low cost
Greater flexibility
Ability to do previously impractical analysis
Where to Start with Big Data Problem Solving Text Mining (unstructured
data that was previously not available)
Pattern Recognition (find previously unknown patterns in the data)
Collaborative filtering (power of the crowd)
Sentiment analysis (Beyond text mining)
Prediction models (new data means new insights about what may come)
Modeling true risk (new data means better forecasts)
Recommendation engines (engage customers)
POS analysis (real-time analysis)
Data “sandbox” (new methods for testing new products concepts)
Data Driven Decision Making Framework – Insights to Action
Source: Social Business By Design Dion Hinchcliffe
Signal Types Signals have attributes depending on their representation in time or frequency domain can also be categorized into multiple classes
All signal types have certain qualities that describe how quickly signals can be generated (frequency), how often the signals vary (rate of change), whether they are
forward looking (quality), and how responsive they are to stimulus (sensitivity)
Rate of Change (Slow or Fast)
Quality (Predictive or Descriptive)
Sensitivity (Sensitive or Insensitive)
Frequency (High or Low)
Sentiment Expressed as
positive, neutral, or
negative, the prevailing attitude
towards and entity
Behavior These signals
identify persistent trends or
patterns in behavior over
time
Event/Alert A discrete
signal generated when
certain threshold
conditions are met
Clusters Signals based on an entity’s
cohort characteristics
Correlation Measures the correlation of
entities against their prescribed attributes over
time
Finding Signals in Unstructured Data High quality signals are necessary to distill the relationship among all the of the Entities across all records (including their time dimension) involving those Entities to turn Big Data into Small Data and capture underlying patterns to create useful inputs to be processed by a machine learning algorithm.
For each dimension, develop meta-data, ontology, statistical measures,
and models Timing/ Recency
Measure the freshness of the data and of the insight
Source Measure sources’ strength:
originality, importance,
quality, quantity, influence
Content Derive the sentiment
and meaning from
tracking tools to
syntactic and semantics analysis
Context Create symbol
language to describe
environments in which the data resides
Clickstreams
Social
Articles
Blogs
Tweets
New Solutions Must Aid Human Insight
Big Data + Amplified Human Intelligence = Better Decisions
Last Decade
- Structured Data - Conclusive Dashboards - Small scale / sampling A data architect built a view to reach a specific conclusion
Next 5 Years
- Any data, from anywhere - Intuitive
exploration - Making sense of it
at scale Business users easily find, explore, visualize and navigate insights
Where to Start
Know Your Ecosystem
Business leaders must know the tools of the trade in order to know what is truly possible.
Data Driven Organizations Always Question the Data
• What business opportunity/problem are we trying to solve?
• What questions do we need to answer to solve the problem?
• What data do we need to answer the questions?
• What data do we have?
• How can data help differentiate us in the market?
• What data is IP for us? Revenue generating for us?
• How do we integrate the right data together?
• How do we manage the quality of the data?
• What data does this relate to (master data)?
• Do we have all the data about this (person, event, thing, etc.)?
• What are the permissible purposes of the data? (compliance, regulatory environment)
• Who is allowed to access the data? Use this data?
Data Driven Spider Graph
Data Driven
Customer Experience
Data Science
Big Data IT
Business Strategists
Business Intelligence
Tools
Social
Traditional IT
Logistic
Customer Care
Always Remember: Data, Insights, Actions
Listen • Listen to the data streams
Share • Share the data with the rest of the organization
Engage • Engage to the data to find the insights
Innovate • Innovate new ideas from the insights gained from the data
Perform
• Perform insightful actions from the data to create better customer experiences
