data quality –“are we there yet?” august 17, 2011...presented by arvind mattoo, cbip data...
TRANSCRIPT
![Page 1: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/1.jpg)
Presented
By
Arvind Mattoo, CBIP
Data Quality – “Are We There Yet?”
August 17, 2011
![Page 2: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/2.jpg)
2
Data Quality
• Data Quality – Explained
• Data Quality – CEO’s Concern
• Data Quality – CIO’s Nightmare
• Data Quality – PM’s Approach
• Data Quality – IT’s Deliverable
![Page 3: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/3.jpg)
3
Data Quality – Dimensions
Data QualityFACT
• Accessible• Consistent• Complete• Lineage• Controllable• Secure
• Accurate• Integral• Unique• Valid• Secure
• Relevant• Existent• Reliable• Reportable• Compliant• Measurable
• Currency• Timeliness• Historical
Process Dimension Business Dimension
Technical Dimension Time Dimension
![Page 4: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/4.jpg)
4
Dimension – Business
Relevant: Does it Map to our Requirements?
Existent: Do we Own it?
Reliable: Can we Trust it?
Reportable: Can we Visualize it?
Compliance: Is it Mandated?
Measurable: Can we Baseline it?
![Page 5: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/5.jpg)
5
Dimension – Process
Accessible: Can I Get it?
Consistent: Can I Standardize it?
Complete: Does it Encompass Usability?
Lineage: Can we Trace it?
Controllable: Can we Discipline it?
Secure: Can we Trust it?
![Page 6: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/6.jpg)
6
Dimension – Technical
Accurate: To what Degree does it Jive?
Integral: Does it Comply Structurally?
Unique: To what extent is it De-Duped?
Valid: Does it Conform by the Rules?
Secure: To what Level is it Secured?
![Page 7: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/7.jpg)
7
Dimension – Time
Currency: To what Degree is it Current?
Timeliness: How Readily is it Available?
Historical: How far back can we Audit?
![Page 8: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/8.jpg)
8
Data Quality – CEO’s Concern
• Lack of Strategic Information Capabilities
• Quality of Decision Making
• Lack of Visibility
• Loss of Opportunities
• Increasing IT Expenditures
• Diminishing Rate of Return
• Lack of Collaboration
![Page 9: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/9.jpg)
9
Data Quality – CIO’s Nightmare
• How did we get into this mess?
• How does it impact our business?
• Are we the only one?
• How do we get out of this?
• How do we sustain it?
• Are we there yet?
![Page 10: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/10.jpg)
10
Data Quality – As We Speak!
• Data Misused: Not Authorized
• Data Abused: Not Qualified
• Data Confused: Not Clarified
• Data Refused: Not Ratified
• Data Diffused: Not Archived
![Page 11: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/11.jpg)
11
How did we get into this mess?
Business
• Mergers
• Acquisitions
• Expansions
• Diversification
• Regulatory
• Lack of Ownership
• Business Process Changes
• Lack of Executive Awareness
• Lack of Training
Technical
• Conversion
• Manual Data Feeds
• Lack of Automation
• System Upgrades
• Consolidation
• Insufficient DQ Rules
• System Errors
• Source System Changes
• Lack of Expertise
![Page 12: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/12.jpg)
12
How does it impact our business?
Surging Cost
• Reputation at Stake• Lower Quality of Service• Customer dissatisfaction• Loss of Motivation • Compliance Issues• Expectations not met
• Time to Reconcile Data• Delay in New System Deployment• Poor System Performance • Loss of Credibility • Downstream System Data Issues• No Single Version of Truth
CEO CIO
![Page 13: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/13.jpg)
13
Are we the only one?
![Page 14: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/14.jpg)
14
How Bad is it?
![Page 15: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/15.jpg)
15
Who is Controlling Whom?
![Page 16: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/16.jpg)
16
How do we get out of this?
• Data Quality – PM’s Approach
• Data Quality – IT’s Deliverables
![Page 17: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/17.jpg)
17
Data Quality – PM’s Approach
• Assess/Profile Data
• Define Baseline
• Define Metrics and Targets
• Define and Build Data Quality Rules
• Enforce Data Standards across Board
• Monitor Data Quality against Targets
• Review Exceptions and Gaps
• Cataloguing Errors
• Refine Data Quality Rules
• Manage Data Quality against Targets
• Automate Data Quality Process
• Fine Tuning Data Quality Rules
Methodology
![Page 18: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/18.jpg)
18
Data Quality – PM’s Approach
• Governance Committee
• Data Stewards
• Business SME
• Business Analysts
• Technology SME
• Process SME
Governance Team
![Page 19: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/19.jpg)
19
Data Quality – PM’s Approach
• Data Profiler
• CRM
• Data Warehouse
• Master Data Management
• ETL/ELT
• CASE
• Custom Data Integration
• Master Data Integration
Technology
![Page 20: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/20.jpg)
20
Data Quality – IT’s Deliverables
• Referential Integrity Rules
• Attribute Rules
• Attribute Domain Rules
• Attribute Dependency Rules
• Historical Data Rules
• State-Dependent Rules
Establish Data Quality Rules
Cataloguing Errors
• Error Tracking• Error Notifications/Alerts
Score carding
• Record Level• Domain Level
![Page 21: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/21.jpg)
21
How do we Sustain over time?
• Follow Data Quality Framework
• Profile Data consistently
• Update Rule Based Engine Frequently
• Exploit Embedded DQ Functions/Solutions
• Adopt Proactive Approach
• Establish Stewardship
• Practice DQ Governance
![Page 22: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/22.jpg)
22
Data Quality – Are We There Yet?
• Accessible
• Relevant
• Reliable
• Reportable
• Compliant
• Accurate
• Consistent
• Complete
• Secured
• Integral
![Page 23: Data Quality –“Are We There Yet?” August 17, 2011...Presented By Arvind Mattoo, CBIP Data Quality –“Are We There Yet?” August 17, 2011](https://reader033.vdocuments.net/reader033/viewer/2022052005/601984508bf28b1b295e7360/html5/thumbnails/23.jpg)
23
Data Quality – Are We There Yet?
Not really!
Data Quality is an iterative process…