the abc of data governance: driving information excellence
DESCRIPTION
Overview of Data Governance requirements, techniques and outcomes. Presented at 5th Annual Records & Information Officers' Forum, Melbourne 19-20 Feb 2014.TRANSCRIPT
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
The ABC of Data Governance and its role in driving Information Management Excellence
Alan D. Duncan February 2014
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
A bit about me....
• Alan Duncan, Director of Data Governance, UNSW
• 21 years Information Management & Business
Consulting
– EDS, KPMG, CPW, Acuma, Pelion, SMS
– Scottish Power, United Distillers, O2, Astra Zeneca,
Carphone Warehouse, Vodafone, Riyad Bank
– Commonwealth Bank, NSW Roads & Maritime
Services, Centrelink, OATSIH, NSW Family &
Community Services, CASA, AMSA, FaHCSIA, DAFF,
Navy…
• Information-Management.com “Top 12 on Twitter”
• Best supporting Actor, 2005 Barnet Drama Festival
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
…and a bit about UNSW.
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Agenda
• Why we need to rethink our Information
Management Strategies
• Techniques and capabilities we need to focus on
• What data should be stored, and how long should
we keep it?
• Information Management Excellence
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Why we need to rethink our information
Management Strategies,
Sponsored by Thomas Edison
“The value of an idea
lies in the using of it.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“A” is for “Action”
Identify measurable and targeted Business Outcomes
Why do we need information? For whom? What will we do
differently?
Establish DG Operating Model
Who is accountable? By what
processes?
Execute Activities & Tasks
How do we deliver? Who does the
work?
Confirm the Information Holdings & Gaps
What do we need to provide? (Content + Context)
Implement DG/IMCC Services
Catalogue:
What core capabilities do we need? “When it is obvious that the
goals cannot be reached,
don't adjust the goals,
adjust the action steps.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“A” is for “Access”
• Stimulus to improve data quality
• Consistency of data definitions
• Openness and trust
• Transparency & accountability
• Opportunity value
• Proactive publication and Open
Data vs. “Need to know”
7
“Publish and be damned!”
http://www.ted.com/talks/tim_berners_lee_the_year_open_data_went_worldwide.html
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“A” is for Analytics
8
“The alchemists in their
search for gold discovered
many other things of
greater value.”
• “Traditional” BI (reporting & ad-hoc analysis)
• Data Mining
• Statistical modelling
• Data visualisation
• Textual analytics
• What questions do we want to answer?
• What questions can we answer with the data
we’ve got?
• What other data would we need?
• What does the data tell us we should be
asking?
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Summary: Information Use Cases • Based on our current understanding of business needs, the following classes of
Information Use Case are identified
• Detailed Requirements Analysis should be conducted on a project-by-project basis
to explore any detailed Use Cases within each class
• Not all detailed Use Cases need to be defined ahead of time
• Solutions should be flexible to accommodate new and changing Use Cases
Structured data
reporting
Strategic Intelligence
and Data Mining
Publish content to a community
Executive briefings
Education, Training, Learning
Search for content
previously created
Records Management, Compliance &
Audit
GIPA & Privacy
Responses
Ability to publish Filtering/screening/validation of what gets published
Feedback loop, measure of usefulness & continuous improvement
Shared understanding (IT & Business)
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Techniques and capabilities,
sponsored by Mark Twain
“To succeed in life,
you need two things:
ignorance and
confidence.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“A” is for “Asset”
Owners
AssetManagement
Tools
Governance
Admin
Experts
User Community
Information
Asset
Steward
OwnersOwners
The “Information Asset Community”
Directives
WMSAIRS
Example High Level Data Systems & Flows
Version 1.0
General
Mandatory
Core
Corporate Support
Automatic One-way Relationship
Automatic Two-way Relationship
Manual One-way Relationship
Manual Two-way Relationship
External
External System
AIRSInterchange
AUSSAR
AME
Cyber Exams
ATSL
CLIC
SDR
Publishing System
GMEL
TRIM AD
FRLI
Web Control Mgmt
System (WCMS)
FMIS
STI
ESIR
Comweb
ATO Business
Portal
Inventory Mgmt
System (IMS)
EPK
ComBIZ Online
ProMaster
FCAT
CalumoCBMS(DoFD)
COMCARE
Thomas Logistics
SM7
HRMS
HRFlex
DTAR/OTAR
APEX
AOD Audit
AOD Case Mgmt
SystemTimelog
eRooms
Symbion Health System
API Upload System
ChangePoint Testing System
MRS
ASSP
AWS
AFD
ASIR
ADMS
TrackerATOG Job Register
AOC Surveys
Industry PaymentsCompensation Payments
Financial Actuals
Financial Actuals
Employee Expenses / Adjustment
Journals
Salary Payments
Cash Payments
Payroll (Salary)
Cash Payments / Organisation Info
Human Resources Finance
Surveillance/Audit/Reporting/Tracking
Workflow and Online Collaboration/Service Delivery
Service Delivery
Service Delivery
Service Delivery, HR & Finance, Agreements, Permissions, Aerodromes, Participants, Aircraft
Medical Examinations
Medical Exams
Surveillance /Audits/ Reporting/ Tracking
Alcohol and Other Drugs
Surveys / Surveillance
Events/Occurrences, Aircraft, Aerodromes
Surveys/ Certifications
Examinations
Work Orders
Surveillance /Audits
Aerodromes
Aircraft
Events/Occurrences, Aircraft, Aerodromes
Defects/Events/Occurrences, Aircraft, Aerodromes
Exemptions Database
Alternative Means of
Compliance (AMOC)
Exemptions
AMOC / Exemptions
Human Resources – Flex
Time
Human Resources - Travel
Physical Inventory
Audit Data
workflow / service delivery
workflow / service delivery
Contacts – Ind, Org’s Contacts – Ind, Org’s
ExaminationsMedical
Examinations
Search and Rescue
Surveys
Human Resources – Time
Aircraft Equipment Finances
MMELBaseline/Minimal Equipment
Medical
PAWS
Retain
Details of Operators
Incidents
Applications / Permissions
Trending
Workflow
MAAT
Permissions / Change of Status
Permissions / Change of Status
Service Log
Alternative Means of Compliance (AMOC)
Dangerous Goods
Dangerous Goods
Content
Inventory
FTTO
FTNS
Individual Flight Data
Organisational Flight Data
Human Resources – Time
Aircraft
Individuals/ARNS
Payments
HR - TimeCash Receipts
Reconcile Invoice against
Flown Hours
Surveys / Surveillance
Enterprise Data Warehouse
CASA Internet
Airports
Landings/ Take Offs
Data
Mandatory
Core
Corporate Support
External
Business Process
Surveillance/Audit/
Reporting/Tracking
Bank Data File
PAYG payments, Salary
payments, and Superannuation
payments.
External Superannuation
Companies
Cash Payments
Superannuation Contributions
Suppliers
Remittance Advice
300+ Access Databases
Contacts
Airspace
Organisational
Human Resources
Aircraft
Permissions Info Asset Register
(inventory) System Interfaces map
“Science is organized
knowledge. Wisdom is
organized life.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
QW007
ORG007
PROC005
PROC003
STRAT007
STRAT002
STRAT005
CUL002
ORG003
ORG006
POL001
POL002
ORG002
SYS004
STRAT015
SYS008
QW004
CUL003
ORG008
QW006
ORG010
PROC014
PROC002
POL003
SYS010
SYS009
SYS002 SYS003
QW005
QW002
SYS006
ORG009
QW003
QW001
SYS007
SYS001
PROC010
PROC008
PROC004
PROC001
PROC011
POL004
POL005
PROC007
PROC009
PROC013
“B” is for “Benchmark”
“One day Alice came to a fork in the
road and saw a Cheshire cat in a
tree. Which road do I take? she
asked. Where do you want to go?
was his response. I don't know,
Alice answered. Then, said the cat, it
doesn't matter.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“B” is for “Big Data”
• Three, four, five, six “Vees”?!
• A lot of data (Tb/day)
• Streaming data (monitoring, flow-of-control and
alerting analytics)
• Inference from semi-structured data (Twitter,
Facebook)
• Synthesise insight from millions of pages of text
• Programmatic analysis for specific scenarios (hard in
SQL)
• A disruptive catalyst to put information at the top of
the organisational agenda
• Not just about the data! Business scenarios are key
• Beware the Vendors!
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“C” is for “Cloud”
14
… but security, privacy considerations are heightened.
In principle, it’s just another place to store data….
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“C” is for compliance
• Freedom of Information Act 1982 (Cth)
• Freedom of Information Amendment (Reform) Act 2010
(Cth)
• Privacy Act 1988 (Cth)
• Privacy Amendment (Private Sector) Act 2000
• Privacy Amendment Act 2012 (Cth)
• Privacy Amendments (Privacy Alerts) Bill 2013 (Cth)
• State Records Act 1998 (NSW)
• Government Information (Public Access) Act 2009 (NSW)
• Privacy & Personal Information Protection Act 1998
(NSW)
• Health Records & Information Privacy Act 2002 (NSW)
• NSW Government Guide To Labelling Sensitive
Information 2011 (NSW Financial & Services)
15
But is “compliance” a motivator?
“All I want is compliance with
my wishes, after reasonable
discussion.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“C” is for Collaboration Culture
16
• A general willingness to share
information
• Co-operative, communicative &
collegiate OR control, coercion
& criticism?
• The “whose data is this?” cue
• Call-to-action?
• Accountability &
measurement?
“Respond intelligently even
to unintelligent treatment.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“D” is for Data Models
Metadata
Repository
Master Data Repositories UNSW Core Systems
Information
Asset
Register
Physical Instantiations
Physical Layer
Logical Layer (Transition)
Analytical
DB Models
Cubes
Conceptual Layer (Business)
Physical
Messages
Formats
DWHDB
HRDB
Student Admin
etc...
Operational
DB Models
Reference models
Data Subject Areas
Data Entities
Data Attributes
Information
Concepts
Business Content Business Rules Data
Business Data
Element
Domain Values
Endorsed Standards
for Content
Business Constraints
Business Measures
Master data models
Classification Entity
Hierarchies
Mappings
Business Rules
Definitions
Business Constraints
Business Measures
Core
SystemsMDM
Me
tad
ata
Ma
na
ge
me
nt P
rocess
Info
rma
tion M
od
el M
an
ag
em
en
t Pro
ce
ss
Info
rmatio
n A
lliance
s: D
ata
Ow
ne
rsh
ip &
Ste
wa
rdship
Pro
cess
MDM Processes
Related Data Governance Processes
Application Logical Data
ModelsLogical
Message
SchemasMDM Data
Model
Systems Data Models
SOA/EP
MessagesG/L
Application Logical Data
ModelsLogical
Message
Schemas
Analytical
DB Models
Cubes
Physical
Messages
Formats
Operational
DB Models
Business Glossary
Conceptual Model: Groupings & Relationships
Data Elements, Definitions,
Aliases, and Security
Data Domains
Enterprise Information Model
“Do not quench your
inspiration and your
imagination; do not
become the slave of your
model.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Information
Disconnect
Careless data entry
& lack of validation
Teams use different
IT systems
?
Organisations
change rapidly
Teams have different ways
of reporting data
Month Region
Multiple codes exist
for the same thing
Data is in different
Formats
Overlapping subsets
in different places
Multiple, inconsistent
master data
“D” is for Data Quality
“Get your facts first,
then you can distort
them as you please.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Summary: Data Governance increases utility
& value of information
Information System
Data Quality Management(Profiling, root-case analysis, issues tracking & resolution)
Data Modelling(Consistent, inter-operable data structures & semantic meaning)
Information Requirements & Business Analysis
(Identification & traceability of business definitions & rules)
Information Asset Register (Catalogue of data holdings)
Information SystemInformation System
Information System(s)
Data Set
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
What data should be stored, and how
long should we keep it?
Sponsored by Freddie Mercury
“The bigger the better;
in everything.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“E” is for “Everything”
• All of the data, all of the time?
• Granular, forensic history
• Modern data management & analytics solutions can make “all
of the data, all of the time” a reality
• The bigger challenge is that the business community is not
analytically skilled enough to navigate the data and draw
meaning from it…
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“E” is for “Expectations”
• Finding data that makes an impact
• Having data for the problem at hand
• Trusting the data to guide your
decision
• Justifying pre-determined answers
• Setting inappropriate goals
• Not having the right data tools
• Not thinking about value
“Two things are infinite.
The universe and human
stupidity. …and I’m not so
sure about the universe.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“F” is for “Forethought”
• Think about both current and future demand
– Cf. Google philosophy to “keep everything”
– Every click, every font change…
• Tiered storage technology for different scenarios
– Fast-access disk for computational processing (TB)
– Drill-through near-line disk (TB/PB)
– Off-line active store (optical, tape) for archive (PB/EB)
• Information Lifecycle Management
– Copy/replication management
– Test data management
– Auditability of data movements between environments
– e.g. IBM Optim, Oracle ILM, Informatica ILM, Hadoop Project
Falcon
• Integration/coupling of HDFS & EDW/BI?
• Data Requirements Management…
“Forethought we may have,
undoubtedly, but not
foresight.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Summary: Data Requirements Management -
two modes of operation
Source new data; Collection & Integration;
Preparation & Quality.
Demand-oriented Inbound requests for specific requirements
Data Factory
(“push”)
Product-based
delivery (“pull”)
Need both “push” and “pull” modes
for evidence-based decision-making
Plus metadata management, info asset management,
data management, process management practices..
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Information Management Excellence
sponsored by Carl Sagan
“I try not to think with my
gut. If I‘m serious about
understanding the world,
thinking with anything
besides my brain, as
tempting as that might be,
is likely to get me into
trouble.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
“G” is for Governance
• “I don’t know what you’re going to do with my data once
you have it.”
• “If I give you my data, you might then ask me to do some
extra work to meet your additional requirements.”
• “You may not interpret the data in the same way that I do.”
• “I’m an expert in this area, you’re not. The data is too
complex for you to understand.”
• “It’s too difficult to get the data out of the system and I’d
need help from I.T.”
• “I don’t have the budget to pay for your requirements.”
• “I’d like to help but I’m just far too busy.”
• “I know there are flaws in the data, but it’s good enough
for my needs. You might criticize me for the errors.”
• “Management may ask additional questions and hold me
to account for the work I’m doing”.
26
“I’m not interested in
preserving the status quo;
I want to overthrow it.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Data Governance: overcoming “I object!”
27
• Control, structure, discipline &
compliance? OR Advocacy service &
information broker?
• Intimate understanding of business goals
& processes
• Engagement, diagnosis & facilitation
• Understand & articulate the meaning of
data, in context
• Coach, mentor and advocate
• Highly visible point-of-access
• Self-service Information Portal
• Conduit, communicate & co-ordinate
• Leadership & direction
• “Info as a Product”
“The art of government is to
make two-thirds of a nation
pay all it possibly can for the
benefit of the other third.”
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Conclusion: Information Excellence EIM Framework:
Enterprise Information Management Framework describes each aspect of an organisations
information management state, provides a baseline of maturity against best practice and a
framework of business transformation to your aspirational information management state.
Provides linkage and balance between business,/IT, and human/technical aspects of EIM.
Information
Governance
Information Security
Information Asset
Mgmt
Metadata
Ownership &
Stewardship
Information and IM
Strategy and
Planning
Information and IM
Quality Mgmt
Information Asset
Classification
Intellectual Property
Reporting design AnalyticsInformation
Security Policy
and Governance
Asset
Management
Human Resources
Security
Management
Knowledge Transfer
Data Mining
Data WarehousingBusiness Intelligence
Information IM
Workforce
Management
Information and IM
Risk Management
Registration
Data Modelling
Data management
Data Integration
Data Cleansing
Data Capture
Data Migration
Data De-duplication
Record Keeping
Knowledge Management
Information Asset Access and Use
Management
Privacy Publishing
CopyrightPhysical and
Environmental
Management
Communications
and Operations
Management
Information
Security Incident
Management
Access
Management
Information
system
acquisition,
development and
maintenance
management
Compliance
Management
Information and IM
Policy, Principles
and Architecture
Information and IM
Governance
Processes
Meta Knowledge
Search and Discovery
ExchangePricing
Licensing and
Rights
Management
Assess and
Accessibility
Redress MechanismsData Quality and
Integrity
Data Conversion
& Transformation
Record Management Archiving Conservation and
Preservation
Record Creation
and Capture
Digital Continuity
Collection Management
Retrieval and Access
Retention and Disposal
Business
Continuity
Enterprise Information Model IM Solutions and Technology IM Policies
Organisation and People Data Governance Information Culture
IM Processes
Business Processes
DB Models
Definitions, Derivations, Decision Rules, Execution Rules
IM Governance Process
IM Stewardship Process
Technical MetaData Management
Logical Model
ETL SpecsReport
DefinitionsSemantic Specs
Data
Marts
ETL Cubes
Semantic
Layer
Sta
ndard
Report L
ibra
ry
ETLOperational
SystemStaging Warehouse
Conceptual Model
Logical Model
Physical Model
Capture & Formalise
Requirements
& Rules
Impact
Assessment
& Implementation
Metadata
Lineage
Impact
Etc.
Metadata
Collection
Asset Alignment/Mgt
Architecture Changes
Architecture
Mgt
A holistic, data-centric approach to Information Management & Data Governance,
addressing both human and technical factors in both Business and IT domains
Alan Duncan, Director of Data Governance, UNSW
E: [email protected] Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Intellectual curiosity
Skeptical scrutiny
Critical thinking
http://www.informationaction.blogspot.com.au/
@Alan_D_Duncan
http://www.linkedin.com/in/alandduncan