data quality –“are we there yet?” august 17, 2011...presented by arvind mattoo, cbip data...

Presented

By

Arvind Mattoo, CBIP

Data Quality – “Are We There Yet?”

August 17, 2011

2

Data Quality

• Data Quality – Explained

• Data Quality – CEO’s Concern

• Data Quality – CIO’s Nightmare

• Data Quality – PM’s Approach

• Data Quality – IT’s Deliverable

3

Data Quality – Dimensions

Data QualityFACT

• Accessible• Consistent• Complete• Lineage• Controllable• Secure

• Accurate• Integral• Unique• Valid• Secure

• Relevant• Existent• Reliable• Reportable• Compliant• Measurable

• Currency• Timeliness• Historical

Process Dimension Business Dimension

Technical Dimension Time Dimension

4

Dimension – Business

Relevant: Does it Map to our Requirements?

Existent: Do we Own it?

Reliable: Can we Trust it?

Reportable: Can we Visualize it?

Compliance: Is it Mandated?

Measurable: Can we Baseline it?

5

Dimension – Process

Accessible: Can I Get it?

Consistent: Can I Standardize it?

Complete: Does it Encompass Usability?

Lineage: Can we Trace it?

Controllable: Can we Discipline it?

Secure: Can we Trust it?

6

Dimension – Technical

Accurate: To what Degree does it Jive?

Integral: Does it Comply Structurally?

Unique: To what extent is it De-Duped?

Valid: Does it Conform by the Rules?

Secure: To what Level is it Secured?

7

Dimension – Time

Currency: To what Degree is it Current?

Timeliness: How Readily is it Available?

Historical: How far back can we Audit?

8

Data Quality – CEO’s Concern

• Lack of Strategic Information Capabilities

• Quality of Decision Making

• Lack of Visibility

• Loss of Opportunities

• Increasing IT Expenditures

• Diminishing Rate of Return

• Lack of Collaboration

9

Data Quality – CIO’s Nightmare

• How did we get into this mess?

• How does it impact our business?

• Are we the only one?

• How do we get out of this?

• How do we sustain it?

• Are we there yet?

10

Data Quality – As We Speak!

• Data Misused: Not Authorized

• Data Abused: Not Qualified

• Data Confused: Not Clarified

• Data Refused: Not Ratified

• Data Diffused: Not Archived

11

How did we get into this mess?

Business

• Mergers

• Acquisitions

• Expansions

• Diversification

• Regulatory

• Lack of Ownership

• Business Process Changes

• Lack of Executive Awareness

• Lack of Training

Technical

• Conversion

• Manual Data Feeds

• Lack of Automation

• System Upgrades

• Consolidation

• Insufficient DQ Rules

• System Errors

• Source System Changes

• Lack of Expertise

12

How does it impact our business?

Surging Cost

• Reputation at Stake• Lower Quality of Service• Customer dissatisfaction• Loss of Motivation • Compliance Issues• Expectations not met

• Time to Reconcile Data• Delay in New System Deployment• Poor System Performance • Loss of Credibility • Downstream System Data Issues• No Single Version of Truth

CEO CIO

13

Are we the only one?

14

How Bad is it?

15

Who is Controlling Whom?

16

How do we get out of this?

• Data Quality – PM’s Approach

• Data Quality – IT’s Deliverables

17

Data Quality – PM’s Approach

• Assess/Profile Data

• Define Baseline

• Define Metrics and Targets

• Define and Build Data Quality Rules

• Enforce Data Standards across Board

• Monitor Data Quality against Targets

• Review Exceptions and Gaps

• Cataloguing Errors

• Refine Data Quality Rules

• Manage Data Quality against Targets

• Automate Data Quality Process

• Fine Tuning Data Quality Rules

Methodology

18

Data Quality – PM’s Approach

• Governance Committee

• Data Stewards

• Business SME

• Business Analysts

• Technology SME

• Process SME

Governance Team

19

Data Quality – PM’s Approach

• Data Profiler

• CRM

• Data Warehouse

• Master Data Management

• ETL/ELT

• CASE

• Custom Data Integration

• Master Data Integration

Technology

20

Data Quality – IT’s Deliverables

• Referential Integrity Rules

• Attribute Rules

• Attribute Domain Rules

• Attribute Dependency Rules

• Historical Data Rules

• State-Dependent Rules

Establish Data Quality Rules

Cataloguing Errors

• Error Tracking• Error Notifications/Alerts

Score carding

• Record Level• Domain Level

21

How do we Sustain over time?

• Follow Data Quality Framework

• Profile Data consistently

• Update Rule Based Engine Frequently

• Exploit Embedded DQ Functions/Solutions

• Adopt Proactive Approach

• Establish Stewardship

• Practice DQ Governance

22

Data Quality – Are We There Yet?

• Accessible

• Relevant

• Reliable

• Reportable

• Compliant

• Accurate

• Consistent

• Complete

• Secured

• Integral

23

Data Quality – Are We There Yet?

Not really!

Data Quality is an iterative process…