questionnaire design, pretesting and quality control in clinical research celia p. kaplan division...

Questionnaire Design, Pretesting and Quality Control in Clinical Research

Celia P. KaplanDivision of General Internal Medicine

August 24th 2009

Good Clinical Practice (GCP)

“A standard for the design, conduct, performance, monitoring, auditing, recording, analysis, and reporting of clinical trials that provides assurances that the data and reported results are credible and accurate, and that the rights, integrity, and confidentiality of trial subjects are protected”

WHO good clinical practice standards

Quality Control

Quality Control

Minimizes the chance of obtaining faulty data and drawing erroneous conclusions

Quality Control

Minimizes the chance of obtaining faulty data and drawing erroneous conclusions

Fulfills administrative and regulatory obligations

Sources of Data Errors Data acquisition


instrument problems


problems instrument lost data


instrument problems lost data falsified data (made up data, changed )


instrument problems lost data falsified data (made up data, changed)

Data recording


instrument problems lost data falsified data (made up data or changed)

Data recording Data entry


instrument problems lost data falsified data (made up data or changed)

Data recording Data entry Data management

Quality Control Stages Pre-implementation/Developmental Stage


Design high-quality questionnaires/interviews and forms



Pretest all aspects of the project



Pretest all aspects of the project Develop an Operations Manual


Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system


Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification



Implementation



Implementation Make periodic study- and data-monitoring

reports



Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings



Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and

performance reviews



Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification

Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and performance

reviews

Questionnaires/interviews Much of the data in clinical and

epidemiological research is gathered using questionnaires and interviews

Questionnaire Measures

Desired qualities Reliability – the extent to which the

measures give consistent results A reliable measure is consistent, responses do not

change Test-retest reliability Statistical techniques

Validity – the extent to which the measures reflect the “truth”

Survey measures: Validity Face validity: the researcher’s best judgment as

to whether the item is measuring what it is supposed to measure

Predictive validity: the item accurately predicts an outcome (e.g., college entrance exams predicting college performance)

Concurrent validity: the item is validated by comparing it against a known valid measure (e.g., a measure of job satisfaction might be correlated with work performance)

Survey measures: Validity Content validity: the items are a representative

sample of the characteristics that comprise the area they intend to measure. Determination of content validity usually relies on experts Example: Patient decision making style

Construct validity: the item demonstrates that it accurately measures a psychological construct like depression or satisfaction

Questionnaire MeasuresDesired qualities Reliability – the extent to which the measures give consistent resultsValidity – the extent to which the measures reflect the “truth”

Questionnaire MeasuresDesired qualities Reliability – the extent to which the measures give consistent resultsValidity – the extent to which the measures reflect the “truth”

Questionnaire Measures Desired qualities

Accommodate participants’ age, literacy,

cultural, linguistic, and SES characteristics

Yield good variability in answers

Result in low percent of missing data

Questionnaire Development: Pitfalls

Double-barreled questions – Avoid “and/or” statements in questions e.g., “Does your department have a special

recruitment policy for ethnic minorities or women?”

Long questions and/or questionnaires

Use of “other” as a response category

Use of Other as Response Category

Example

10. Are you Latino/a or Hispanic? 1 Yes

0 No

11. What is your race/ethnicity? Please check one answer only. 1 Black or African American 2 Asian, Asian American or Pacific Islander 3 White, European American or Caucasian 4 American Indian or Alaska Native 5 Other Please specify ____________________

Use of Other as Response Category

Example1 Ameriasian 19 INDIAN (EAST) AMERICAN

2 American 20 Italian

3 Arab 21 MIDDLE EAST

4 Ashkenazi 22 Middle Eastern/ Indian Subcontinent

5 Asian and Caucasian 23 Mixed

6 Brazilian 24 NATIVE AMERICAN

7 Cuban 25 Oriental

8 East Indian 26 PERSIAN

9 Egyptian 27 POLISH AMERICAN

10 EGYPTIAN 28 South Asian

11 egyptian american 29 Southern Asian

12 From India 30 West Indian

13 half Latin, half black 31 White, Arab

14 Indain American 32

15 India- Indian 33

16 Indian 34

17 INDIAN 35

18 Indian (east) 36

Questionnaire Development: Order

First question should be a fact

Beginning questions should stimulate interest

Questions should be in logical order and groupings

If ask attitude questions first and then behavior questions, may inflate reports of behavior

Sensitive questions and open-ended questions should be at the end of the survey

e.g., sexual behavior, religion, income

Questionnaire Development: Language Use simple, clear, common, and

natural language Avoid jargon, complex terms, clichés,

colloquialisms, e.g., “down in the dumps” Check number of syllables in words Check reading level: aim for < 8th grade Define ambiguous terms, e.g., “family” or

“neighborhood” Keep sentences short

Questionnaire Development: Recall

Solutions: Narrow the reference period Average the response: ask about a typical

day or in general (e.g., hours of sleep) Use landmark events or milestones to

anchor time frame: “Since you moved to the US,” or use calendar with major holidays to establish major life events

Use cues to generate associations to jog memory: describe procedure, setting

Questionnaire Design

1Select Interview Methodology

Interviewmethodology


2Review existing

questionnaires and measures


Reviewexisting

measures


3Conduct exploratory

qualitative work


Reviewexisting

measures

Conductexploratory

work


4Develop a draft survey


Reviewexisting

measures

Conductexploratory

work

Draft survey


5Translation and back-translation


Reviewexisting

measures

Conductexploratory

work

Draft survey

Translation/back-translation


6Pre-testing


Reviewexisting

measures

Conductexploratory

work

Draft survey


Pre-test the survey


7Finalize the Questionnaire


Reviewexisting

measures

Conductexploratory

work

Draft survey


Pre-test the survey

Finalizedinstrument

Questionnaire design

Selection of interview methodology

Face-to-face or in-person interview

Telephone

Self-administered, paper

Self-administered, computer/web/e-mail


Interview methodology: Face-to-Face

Advantages

Enables the interviewer to establish rapport with respondent

Permits more complex questions Permits the use of visual aids Longer interviews are possible

Disadvantages Cost per interview Quality of answers too personal (sensitive

questions)


Interview methodology: Telephone Advantages

Faster contact of participants Better to elicit personal/sensitive information Results are available right away if using

computer-assisted interviewing (CAI) Can do random selection of telephone numbers

Disadvantages Certain sectors of the population are switching to

cell phones only More expensive than mail surveys Difficulties in reaching participants during the day


Interview methodology: Self-administered and mail surveys (Paper)

Advantages One of the least expensive methodologies Needs addresses of potential participants Allows participants to answer at their leisure Less intrusive Elimination of interviewer bias

Disadvantages Response rate is low Difficulties to develop clear skip patterns It may be answered by a family member Very difficult to implement in low literacy populations


Example: Self-administered

& Mail Surveys


Interview methodology: Self-administered, web-based surveys (Computer)

Advantages Elimination of data entry and editing costs Get better answers to sensitive questions Elimination of interviewer bias Ensure that skip patterns are accurately followed Allows participants to answer at their leisure

Disadvantages Interviewees must have access to a computer Must possess (or purchase) a list of email addresses Lower response rate People may quit in the middle of a questionnaire

Questionnaire designExample: Web-based survey


Survey modalities: Answer types

Numeric open-endedHow old are you? ____

Text open ended How can we improve our services? ______________

Multiple Choice--CategoricalWhat is your marital status

Single 1 Married 2

etc.

Survey modalities: Answer types Multiple choice-- Ordinal

In general, would you say your health is…?1 Excellent2 Very good3 Good4 Fair5 Poor77 DK

Scales On a scale where “10” means you have a great amount

of interest in a topic and “1” means you have none at all, how would you rate your interest in each of the following specialties?

Cardiology ___ Oncology __ Dermatology__


Survey modalities: Answer types Likert-Type scales

Used to know respondents' feelings or attitudes The respondent indicates how closely their feelings

match the question or statement on a rating scale. Response categories: “Strongly Disagree" (least

agreement) to or "Strongly Agree." (most

agreement)

Example: In general, to what degree is each of these factors a

barrier for you in referring a patient to a clinical trial? Not a barrier A major barrier

1 □ 2 □ 3 □ 4 □ 5 □



Review Existing Questionnaires & Measures

Reference databases Medline, Pubmed, Psychinfo, others

Compendia of measures Books that compile various measures

and review their characteristics

Request questionnaires from other investigators


Review Existing Questionnaires & Measures

Is there evidence the instrument works in your target population?

Examine validity and reliability of measures when used in populations similar to your study population

When gold standard does not exist, for construct validity, look at relationship of measure to other measures to see if in hypothesized direction


Conduct Qualitative Exploratory Work

Focus groups or semi-structured interviews Explores a topic before constructing formal

questions Rationale:

Determine what is important Discover how respondents think about the topic Assess better language/words to use Identify response categories


Create a Draft Survey Instrument

Write initial draft Process of revision and refinement through

cycles of translation/back-translation, reviews and pre-tests (Steps 5 and 6)


Translation/Back-translation Procedures

Both versions should have the same meaning Translation and back translation Review by a group of native speakers Challenging when more than one language

involved in the study




reviews

Quality Control

Pre-testing the Questionnaire: Overview

Refines Instruments and questionnaires

Evaluates Recruitment methods

Interventions

Data entry and management system

Protocols

Assists with training of personnel

Quality Control

Pre-testing the Questionnaire

Good to test: Sensitive questions Complex or poorly defined topics Questions with terms respondents may not

understand Translations Layout and instructions

Quality Control

Pre-testing the Questionnaire:Field Testing Techniques

Expert reviews Cognitive interviews Full pretest

Quality Control

Pre-testing the Questionnaire:Expert Reviews Formal and systematic examination of

a questionnaire by experts in the field Can cover question wording, layout,

and skip patterns Fast method prior to formal testing May not be a complete review

Quality Control

Pre-testing the Questionnaire:Cognitive Interview: Overview Diagnostic tool for pretesting survey

questions derived from social and cognitive psychology

Interview about an “interview” Explores the processes by which

respondents reach answers Based on structured questionnaire and

protocol

Quality ControlPre-testing the Questionnaire:Cognitive Interview: Techniques Concurrent proving

Ask probes immediately after respondent has given answer to survey item

Advantage: information is fresh in respondent’s mind Disadvantage: Disrupts interview flow and relationship

between questions Retrospective probing

Ask probes after entire interview or block of questions Advantage: able to assess standard administration of

items Disadvantage: Participant may not remember thought

process

Quality ControlPre-testing the Questionnaire:Cognitive Interview: Techniques Probing questions Think aloud interviews

Interviewer asks respondent to think aloud as they answer question

Paraphrasing Respondents rephrase the question in their own words

Quality Control

Pre-testing the Questionnaire:Cognitive Interview: Key Features Comprehension of the question

Do participants understand words and phrases as intended by the researchers (meaning)

“How did you arrive at your estimate of risk?” “What did you understand by the word ‘risk’?”

Retrieve of information Identify process respondents use to answer questions Examine strategies used to access memory

Edit responses: Decide what to report Are items unacceptable? Is answer embarrassing,

socially undesirable?

Quality Control

Pre-Test

Pretest with a sample similar to the population to be studied

Simultaneously pretest recruitment procedures and questionnaire implementation

Quality Control

Pre-Test: Behavior Coding

Code interaction between interviewer and respondent Examples

R answers before hearing whole question R looks puzzled R asks for clarification I repeats question

Quality Control

Pre-Test: Respondent Debriefing

Structured follow-up questions at the end of the interview to assess: Whether questions were clear Ease of completing the questionnaires If two versions are tested, which is

preferable Whether an important question was

missed

Quality Control

Pre-Test: Interviewer Debriefing Interviewers implement pretest with

participants Interviewers detect problems with

questions, response categories, and skip patterns

Interviewers report issues to investigators

Quality Control

Pre-Test Questionnaire: Summary

Testing takes time and resources Some procedures take place late in the

developmental cycle, limiting the amount of change possible

Danger of making changes that are not further tested

Not testing leads to questionnaires that are potentially unreliable or will give you invalid results




reviews

Quality Control

Operations Manual Aim is to ensure high quality data Clearly written, detailed written instructions to obtain

uniformity across sites, staff, and procedures Standardizes procedures for all aspects research project.

Defines how to… Recruit participants Consent participants Measure variables Transport clinical samples Enter data Monitor progress Conduct data analysis

Requires updates, clarifications, and recording of decisions made

Quality Control

Data Management System Desirable features

Range and field type checks Ease of screen set up and use Double data entry (if possible)

Security features protection of human subjects’ rights (privacy)

For Web-based systems Virus protection to monitor and eliminate security

threats Database server behind firewall All data backed up regularly

Quality Control

Staff Training Training and certification of the research team members

Rules to conduct interviews Rules for recruiting participants, including how to

obtain consent Methods

“Train the trainer” model Audio-visual techniques

Certification/recertification to maintain skills



Implementation Make periodic study- and data-monitoring

reports Conduct periodic project meetings Complete interim staff training and

performance reviews

Quality Control

Data Monitoring Reports

Standard procedures for Identifying missing, incomplete data Identifying incorrect skip patterns Identifying inconsistent and erroneous data Verifying correction

Quality Control

Interim Staff Training and Performance Reviews

Provide continuous supervision Hold frequent meetings of the study team Review all aspects related to implementation of

the study, including Interviewers activities

Listen to interviews Compare measures between interviewers Compare refusal rates between interviewers

Good Clinical Practice

Good Instruments Complete and clear protocol manuals Extensive pretesting Ongoing monitoring procedures

questionnaire design, pretesting and quality control in clinical research celia p. kaplan division...

Documents

data falsified data

faulty data

datafalsified data

operations manualdesign

clinical researchcelia

good clinical practice

reporting of clinical

questionnaire design