reliability and validity of data collection
DESCRIPTION
Reliability and Validity of Data collection. Reliability of Measurement. Measurement is reliable when it yields the same values across repeated measures of the same event Relates to repeatability Not the same as accuracy Low reliability signals suspect data. Threats to Reliability. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/1.jpg)
RELIABILITY AND VALIDITY OF DATA COLLECTION
![Page 2: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/2.jpg)
RELIABILITY OF MEASUREMENT • Measurement is reliable when it yields the same values
across repeated measures of the same event• Relates to repeatability• Not the same as accuracy
• Low reliability signals suspect data
![Page 3: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/3.jpg)
THREATS TO RELIABILITY
1. Human error
• Miss recording a data point
• Usually result from poorly designed measurement systems
• Cumbersome or difficult to use
• To complex
• Can reduce by using technology – Cameras
![Page 4: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/4.jpg)
2. INADEQUATE OBSERVER TRAINING
• Training must be explicit and systematic
• Careful selection of observers
• Must clearly define the target behavior
• Train to competency standard
• Have on-going training to minimize observer drift
• Have back up observers observe the primary observers
![Page 5: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/5.jpg)
3. UNINTENDED INFLUENCES ON OBSERVERS
• Causes all sorts of problems
• Expectations of what the data should look like
• Observer reactivity when she/he is aware that others are evaluating the data
• Measurement bias
• Feedback to observers about how their data relates to the goals of intervention
![Page 6: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/6.jpg)
SOLUTIONS TO RELIABILITY ISSUES
1. Design a good measurement system
• Take your time on the front end
2. Train observers carefully
3. Evaluate extent to which data are accurate and reliable
4. Measure the measurement system
![Page 7: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/7.jpg)
ACCURACY OF MEASUREMENT
• Observed values match the true values of an event
• Issue: Do not want to base research conclusions or treatment decisions on faulty data
![Page 8: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/8.jpg)
PURPOSES OF ACCURACY ASSESSMENT:
• Determine if data are good enough to make decisions
• Discover and correct measurement errors
• Reveal consistent patterns of measurement error
• Assure consumers that data are accurate
![Page 9: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/9.jpg)
OBSERVED VALUES MUST MATCH TRUE VALUES
• Determined by calculating correspondence of each data point with its true value
• Accuracy assessment should be reported in research
![Page 10: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/10.jpg)
INTER- OBSERVER AGREEMENT (IOA) OR RELIABILITY (IOR)
• Is the degree to which two or more independent observers report the same values for the same events
• Used to:
• Determine competency of observers
• Detect observer drift
• Judge clarity of definitions and system
• Increase validity of the data
![Page 11: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/11.jpg)
REQUIREMENTS FOR IOA / IOR
• Observers must:
• Use the same observation code and measurement system
• Observe and measure the same participants and events
• Observe and record independently of one another
![Page 12: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/12.jpg)
METHODS TO CALCULATE IOA / IOR
• (Smaller Freq. / Larger Freq.) * 100 = percentage
• Can be done with intervals as well
• Agreements / Agreements + Disagreements X 100
• Methods can compare:
• Total count recorded by each observer
• Mean count-per-interval
• Exact count-per-interval
• Trial-by-trial
![Page 13: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/13.jpg)
TIMING RECORDING METHODS:
• Total duration IOA
• Mean duration-per-occurrence IOA
• Latency-per-response
• Mean IRT-per-response
![Page 14: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/14.jpg)
INTERVAL RECORDING AND TIME SAMPLING:
• Interval-by-interval IOA (Point by point)
• Scored-interval IOA
• Unscored-interval IOA
![Page 15: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/15.jpg)
CONSIDERATIONS IN IOA
• During each condition and phase of a study
• Distributed across days of the week, time of day, settings, observers
• Minimum of 20% of sessions, preferably 25-30%
• More frequent with complex systems
![Page 16: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/16.jpg)
CONSIDERATIONS IN IOA
• Obtain and report IOA at the same levels at which researchers will report and discuss it within the results
• For each behavior
• For each participant
• In each phase of intervention or baseline
![Page 17: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/17.jpg)
OTHER CONSIDERATIONS
• More conservative methods should be used• Methods that will overestimate actual agreement should
be avoided• If in doubt, report more than one calculation• 80% agreement usually the benchmark
• Higher the better• Depends upon the complexity of the measurement
system
![Page 18: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/18.jpg)
REPORTING IOA
• Can use
• Narrative
• Tables
• Graphs
• Report how, when, and how often IOA was assessed
![Page 19: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/19.jpg)
VALIDITY• Many types
• Are you measuring what you believe you are measuring
• Ensures the data are representative
• In ABA, usually measure:
• a socially significant behavior
• dimension of the behavior relevant to the question
![Page 20: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/20.jpg)
THREATS TO VALIDITY
• Measuring a behavior other than the behavior of interest• Measuring a dimension that is irrelevant or ill suited to
the reason for measuring behavior• Measurement artifacts
• Must provide evidence that the behavior measured is directly related to behavior of interest
![Page 21: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/21.jpg)
EXAMPLES
• Discontinuous measurement
• Poorly scheduled observations
• Insensitive or limiting measurement scales
![Page 22: Reliability and Validity of Data collection](https://reader036.vdocuments.net/reader036/viewer/2022081501/5681352a550346895d9c92ab/html5/thumbnails/22.jpg)
CONCLUSIONS
• Reliabiltiy and validity of data collection are important
• Impacts the client,
• Impacts your reputation for good work