data quality metrics-considerations for achieving trusted...
Post on 23-Mar-2020
4 Views
Preview:
TRANSCRIPT
©Petroliam Nasional Berhad (PETRONAS) 2014 1
© 2014 PETROLIAM NASIONAL BERHAD (PETRONAS)
All rights reserved. No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise) without the permission of the copyright owner.
Data Quality Metrics –Considerations for Achieving Trusted Data
Open
Philip Lesslar
Technical Assurance, Group Technical Data, PETRONAS
Digital Energy Conference
5th October 2016
Impiana Hotel, Kuala Lumpur
©Petroliam Nasional Berhad (PETRONAS) 2014 2
• Trustworthiness and the need for Trusted Data
• Data Quality Metrics (DQM)
• How far can we go with DQM? – The Expert Factor
• So how do we get to trusted data?
• The Quality Data Inventory
• Packaging quality data – The building blocks
• Data delivery and Data Access and Retrieval (DART)
• Quantified benefits and sustainability
• Summary
Contents
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 3
Problem Statement
1. Data collection consumes about half the time in business activities.
– 52% in 10 workflows studied.
2. Everyone who acquires data spends time re-checking it.
– 58 man years/year for well header, check shot, deviation & basic logs (find & quality check).
3. Data come from various sources.
Trustworthiness and the Need for Trusted Data
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 4
Our 5-step Approach Towards Trusted Data…
1 2 3 4 5
Databank
• Single source of truth for each data type
• Identified mandatory and optional attributes needed for business activities
• Standard and approved process for quality checking (QC) of the data
• Identified Data Custodians accountable for ensuring data capture, availability and longevity
• Business appointed Subject Matter Expert (SME) to advise and QC
• Subject Matter Focal (SMF)
• Audit trail & documentation
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 5
It is about measuring data quality…
Data Quality Metrics (DQM) Simplified
E.g. well header must have total depth defined.Business rules
Encoded metrics in application
Data Quality Dashboard
*IQM screenshot (courtesy of Exprodat Inc.)
Features:- Automated rule checking- Consistent- Progressive- Targeted
1
2
3
Total records polled
Total errors found
Quality percent
Trends : Monthly, Weekly, Daily
(1-(1059/141485))*100 = 99.25
Usually written as sequel (SQL) queries or internal proprietary syntax.
User friendly traffic light dashboard allowing for intervention actions.
*IQM = Information Quality Metrics
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 6
Business rules
This arrow represents all the quality checking that a wellheader record needs for it to be fit for business purposes.
70%-90%
Checks:• Mandatory attributes• Data quality dimensions – completeness,
duplication, conformity, consistency, integrity, accuracy
Checks:• Relevant control documents• Significance of missing values• Referential integrity• QC & assurance• Fit for purpose• Assignment of quality
indicator
Subject Matter Expert
Fit-for-purpose data, of known
quality
Given any data type (e.g. well header)…
How Far Can We Go with Data Quality Metrics?
Open
© 2016 Petroliam Nasional Berhad (PETRONAS) 7Open
OriginalFormat Data
Reference Data/Metadata
Master Data/Corporate
“Single Source of Truth”
Derived Data Data Collections
Raw SeismicRaw Logs
Units of measure- Linear measures- Pressure
Static (hard) data- Well header- Deviation- Checkshot- Temperature- Pressure
Processed data- Seismic deconvolution- Seismic filtering- Seismic processing- Edited logs- Spliced logs
Composite data- Completion log- Mud log- Paleontological
composites- TRAPIS
Abbreviations- TD, DFE, KB etc
Interpreted (soft) data- Geological markers- Seismic horizons
Interpreted data- Geological markers- Seismic horizons
Data hoards- Projects en masse- Personal stores- Team folders
Valid Lists Data archive- Projects en masse
Range indicators
Comments
Requires:- Official data repository
Requires:- Standards- Implementation
across all impacted tools and databases
Requires:- Clear processes, workflows
and checkpoints- Proper & official repository- Management and security
processes around repository and data access
Requires:- Standard workflows- Standard algorithms- Standard processes- Housekeeping procedures
Requires:- Standard display and
formatting templates- Procedures
Secondary DataPrimary Data
Data Classification – Digital Data
© 2016 Petroliam Nasional Berhad (PETRONAS) 8
DQMData Quality Metrics
QDIQuality Data Inventory+
So How Do We Get To Trusted Data?
Business rules
70%-90%
Subject Matter Expert
Fit-for-purpose data, of known
quality
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 9
Checkshots
Well Integrity
3D Models
Daily Operations Report (Facilities)
Pressure Volume Temperature (PVT)
Deviation Data
Pipelines
Well Test
Projects
Data QC exercise
Data submissions
Area of strict & structured checking and management.
Well Schematics
The QDI Dashboard
The QDI Dashboard shows,for the important data typesrequired by the business, thelevel of quality checked, fit-for-purpose data that is in theinventory and available forimmediate use.
etc.
Collective of fit-for-purpose quality checked data, required for business activities, and that can be re-used in the company.
What is the Quality Data Inventory (QDI)?
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 10
Filling up the tanks with Trusted Data
Checkshot
Well H
eader
Devia
tion
Well L
ogs (
8)
t1
Checkshot
Well H
eader
Devia
tion
Well L
ogs (
8)
Well I
nte
grity
t2
Pip
elines
Checkshot
Well H
eader
Well L
ogs (
8)
Well I
nte
grity
Pip
elines
tn (future end state)
Schem
atics
Wellte
st
All important data that is required for the business is
properly checked, with signoffs by appropriate
authority.
Over time, the data gets better and better, progress
can be measured and targets can be set.
Amount of quality data increases.
Devia
tion
To provide a structured, consistent and transparent way of showing and demonstrating progress in addressing quality requirements in our data.
What is the Purpose of the Quality Data Inventory?
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 11
Quality Data Envelope
Packaging Quality Data - The Building Blocks
Open
Exploration workflow
Facilities workflow
Production Geology workflow
Analytics workflow
Data Type Building Blocks along the Exploration & Production (EP) Value Chain
PRIMARY DATA SECONDARY DATA
©Petroliam Nasional Berhad (PETRONAS) 2014 12
Data Delivery Methods
Open
Data in flat files,Loadable into selected
applications
Data pull via plug-ins directly into selected
applications
Near-line robotics
Access from bulk-loaded project workspaces
Official DatabanksThe User
EnvironmentData Delivery Mechanisms
EP D
ata
Types
The amount of Trusted Data will increase over time
©Petroliam Nasional Berhad (PETRONAS) 2014 13
Data Delivery – The DART project
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 14
Quantified Benefits And Sustainability
Open
• Quantify problems:− 58 manyrs/yr
NPT.− 52% data
downtime.
• Identify causal factors:
− No framework.− Fragmented
data.− Unclear
inventory.− Unknown quality.− Access
difficulties.− Long delivery
time.
What problems are we trying to solve?
• Prioritize key issues.
• Develop framework and end-to-end working model across key data types.
• Deliver data as early as possible for clarity and transparency.
• Secure additional commitments.
Develop & implement the
solutions
• Quantification methodology.
• Ensure capture of initial states in quantified terms.
• Start calculating deltas as soon as solution in place.
Measure how much of the problem has
been solved
• Track continuous progress via register.
• Ensure broad awareness of progress for continued
momentum.
Implement continuous tracking
to move the dot
©Petroliam Nasional Berhad (PETRONAS) 2014 15
• Data Quality Metrics go a long way in improving data quality but not to trusted level.
– We have our Data Quality Metrics (DQM) programme and are running daily ~1000 queries against about 28 database instances.
– We also have Quality Data Inventory (QDI) which combines DQM with SME checks to bring data to trusted levels. This is tracked via an automated dashboard.
• A database filled with quality data is of no value until it is easily accessible.
– Our Data Access and ReTrieval (DART) project allows users to find and retrieve data quickly.
• We need to track progress and value of quality data.
– Level of quality data is automatically tracked in dashboard for hands-free transparency
– We have implemented a methodology to calculate and track quantified benefits.
• Sustainability is a function of good processes, transparency, quality and accessibility.
Summary
Open
©Petroliam Nasional Berhad (PETRONAS) 2014 16
Thank you
Open
top related