Improving Investigator Adherence to Best Practice Guidelines in

Electronic Data Capture Using an Automated Tool (CheckMate)Swan Ellert MS, Linda Carlin PhD, Umit Kaya MS, Jessica Bondy MHA, Michael G. Kahn MD, PhD

Colorado Clinical & Translational Sciences Institute, University of Colorado Anschutz Medical Campus

Electronic data capture (EDC) best practice guidelines for

database design are intended to improve quality of

collected data by reducing risk of data entry errors. EDC

applications provide features to support many of these best

practices, such as minimizing use of free text entry,

validating text field formats, and setting expected

minimum/maximum values.

Background

Problem Description

Colorado Clinical & Translational Sciences Institute Aurora, CO cctsi.ucdenver.eduSupported by NIH/NCRR Colorado CTSI Grant Number 3UL1RR025780-02S3. Its contents are the authors’ sole responsibility and do not necessarily represent official NIH views.

CheckMate Features

CheckMate is a Microsoft Excel macro developed to identify

areas in a REDCap database data dictionary that do not

conform with applicable best practice guidelines. The

specific guidelines the tool currently identifies are shown in

the central panel. Most CheckMate recommendations are

made by searching for common words (e.g.. date, phone,

email) and matching these words to a guideline.

The percent of free form text did not

change across all studies.

Text validators were already applied

99% of the time across all 9 studies

that did not implement CheckMate

recommendations. For the 5 studies

that did implement CheckMate

recommendations, the text validator

percent went from 73% to 76% across

all studies.

Min/Max values on appropriate text

fields remained low (under 40%) across

all studies

PHI identification made the biggest

gain. However, PHI identification

remained under 50% across all

studies. Further analysis needs to be

done to determine whether word

searches correctly identify PHI.

Example CheckMate Data Dictionary Scan

and User Summary Output

The Colorado Clinical and Translational Science Institute

currently encourages researchers to use REDCap as their

data collection tool. This policy allows investigators, who

understand their study data needs best, to create forms

and determine what fields need to be assigned within the

forms. However, because researchers may not be familiar

with data management best practices, these practices can

often be ignored or misunderstood by investigators. We

created a tool to quickly review database designs for

adherence to best practices, and to provide

recommendations to investigators for improving their EDC

forms prior to data collection.

Create instructions to investigators about how to prepare a data

dictionary for upload to CheckMate (columns to remove, save format,

etc.)

Move “Best Practice Suggestion” column to far right beside “Best

Practice Followed”

Reorganize “Specific Recommendations” to mimic workflow, e.g.,

put text validation suggestions (4, 5) before min/max suggestions (2)

Suggest “Choices” option as well as text validator

Target “Text Validation Min” and “Text Validation Max” separately

Tighten words searches, to lower misidentified best practices (e.g.

search on “Name” identifies “Name” and “Name of Data Entry

Person”)

Check for “Choices” where multiple codes have the same meaning

(i.e., both 99 and 98 mean “Unknown” in the same study data

dictionary)

Future Directions

Evaluate whether other changes, not specifically suggested by

CheckMate, are made to the data dictionary as a result of reviewing

the CheckMate scan

Observe whether reviewing and/or implementing CheckMate

results limits the number of post-deployment changes

Review post-deployment changes to see whether they relate to a

problem identified but not implemented by the investigator in the

CheckMate scan

Validate specifically labeled fields against standard data

definitions. For example, verify that a field label of “Ethnicity” maps

to Cancer Biomedical Informatics Grid (caBig) permissible values.

Present CheckMate as opportunity to REDCap Consortium for use

by other consortium members or for future REDCap enhancements

CheckMate Improvements

Initial Findings

References:

1. CDISC CDASH Core and Domain Teams. Clinical Data Acquisition Standards Harmonization (CDASH); October 2008.

2. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--A metadata-driven

methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics.

2009;42:377–381.

Best Practice

Suggestion Variable or Field Name Field Label

Choices OR

Calculations Text Validation Type Text Validation Min Text Validation Max Identifier? Best Practice Followed

studid

Study Identification

Number

1,2 mrn

Medical Record

Number number 4

1 last_name Last Name

1,2 dob Date of Birth date 4

sex Gender 0, Female | 1, Male

4,5 weight

Weight at time of

surgery

2 surgdate Date of surgery date 4

General Recommendations:

16% of the fields in this study are free form text. If responses can be categorized, consider using a dropdown field type to reduce risk of data entry error and make the data easier to analyze.

3 out of 8 forms in this study have more than 30 fields. Consider creating shorter forms for better data entry.

Specific Recommendations (not all recommendations listed below will apply to this study):

1. Possible PHI. Consider placing a 'Y' in column M [Identifier?]

2. For text fields validated as date, integer, or number, consider entering minimum and maximum expected values in columns K [Text Validation Min] and L [Text Validation Max] to decrease risk of data entry error.

3. Statistical results will vary when using dynamic expressions (e.g. 'today') in calculations. Consider using a fixed date, e.g. 'date of enrollment'.

4. When using text fields, consider validating the field to expect a specific data type by entering it in column J [Text Validation Type]. Available options are: date, time, integer, number, zipcode, phone and email.

5. When entering common lab/physical exam values, consider validating the text field to expect number or integer values in column J [Text Validation Type]

6. Consider making street, city, state and zip separate fields in the database

Fields in green meet best practice guidelines

CheckMate Scan Workflow

PI drafts DDPre-Deployment CheckMate Scan

CheckMate Scan recommendations

sent to PI

PI decides which recommendations

to acceptStudy goes live

Post-Deployment CheckMate Scan

0 10 20 30 40 50 60 70 80 90 100

% possible PHI fields

% Validated Free Form Text with Min/Max

% Free Form Text (Validated)

% Free Form Text

Post-Deployment Scan (5 studies with CheckMate scans implemented)

Pre-Deployment Scan (5 studies with CheckMate scans implemented)

Pre & Post-Deployment (9 studies with no CheckMate recommendations implemented)