1 the good, the bad, and the ugly: collecting and reporting quality performance data
TRANSCRIPT
2
Presentation Plan
Data definition
Uses of Data
Data Quality Assessments (DQAs)
5 Elements of DQA
Practical Tips for Assessing Data Quality
3
What is Data?
Data is observed and recorded information
If it is not in writing, it does not exist
4
Data Can be Used to .. Make important decisions Ask questions such as:
Were targets met?
How did performance comparewith last year?
What is the trend fromthe baseline year?
How does actual datacompare with targets?
5
Data Can be Used to…
Address Implementation problems Are the activities on track and outputs on time?
Are outputs being produced in sufficient quantity?
Is the quality of the activities and outputs adequate?
Are our partners achieving critical results and outputs for which they are responsible?
6
Data Can be Used to….
Share results widely
Hold individuals and partners accountable
Encourage and emphasize activities that contribute to results
8
Data Quality Assessments Are…. Procedures for determining if a data set is
suitable for its intended purpose.
It determines data’s “fitness for use”
9
How Good Do Data Have to Be?When deciding about data quality, ask:
What level of quality is needed for confident decisions?
How critical are the data for decision-making? What are the consequences of decisions based on
the data?
10
The Elements of Data QualityValidity— Do data clearly and directly measure what is
intended?
Reliability— Can the same results be replicated using the same measurement procedures?
Timeliness— Are data sufficiently current and available for decision-making?
Precision— What margin of error is acceptable?
Integrity— Are mechanisms in place to protect data from manipulation for political or personal reasons?
11
Element 1: Validity
“Data are valid to the extent that they clearly, directly and adequately represent the result it was intended to measure”
12
Face validity:
Is simply looking at the concept to see if it makes sense, on the face of it.
If it looks like a duck and quacks like a duck,
then it must be a duck!
13
Construct Validity:
The extent to which a measure assesses specific traits or concepts (e.g IQ)
More rigorous test of validity than face or content validity
Instruments with high construct validity are used to test theory:
Content Validity:The extent to which a measure represents all facets of a social concept (eg. self-esteem, ethnic identity, attitudes).
14
Validity Questions 1: Are samples representative? Are the instruments/ instructions/questions
clear? If translated, are instrument questions
equivalent and culturally appropriate? Were steps taken to limit transcription errors?
Were the data collectors trained?
For raw data, were computation/compilation formulas written down and consistently applied?
15
Reliability:
“Data should reflect stable and consistent data collection processes and analysis methods over time. Progress toward performance targets should reflect real changes rather than variations in data collection methods.”
Are the same sampling and data collection methods, procedures, and instruments used from year to year, location to location, and among contributing IPs?
Are there internal data quality control procedures
written down and followed? Are data collection, computation, analysis and reporting procedures
written down and followed?
16
Reliability Test
Can an independent researcher obtain the same results from thesame raw data?
Are written internal data quality control procedures in place and followed?
Are data collection, computation, analysis andreporting procedures written down and followed?
17
Timeliness:
Are the data timely (current and available on a regular basis)?
Is the timing good (available when needed) for the decisions and/or reports they support?
18
Precision: Are the data of sufficient detail support decision-
making?
Are data accurate enough to present a fair picture of performance?
If a survey was used, was the margin of error calculated? Is the margin of error acceptable to decision makers? Are the data within the margin of error?
Remember: The smaller the margin of error, the greater the cost of data collection.
19
Integrity:
Are mechanisms in place to prevent corruption of the datafor personal, political, or professional interests?
Has an independent DQA been done? Do the data pass the “burden of proof” test?
(They pass unless manipulation can be proven.)
20
In summary, A DQA Answers…
How good your data is for audit If you need further
disaggregation for meaningful use
How will we use the data How aware IP are about data
strengths and weaknesses
21
Practical Tips for Assessing Data Quality Build assessment into
normal work processes Use software checks & edits
of data on computer systems
Get feedback from users of the data
Compare the data with data from other sources
Obtain verification by independent parties