Sources of bias in research

Your bait of falsehood takes this carp of truth;

And thus do we of wisdom and of reach,

With windlasses and with assays of bias,

By indirections find directions out.

-Wm. Shakespeare (“Hamlet”)

Scientists’ attitudes

• There is always a degree of uncertainty;

• Important to remain open-minded;

• Everyone is fallible;

• Must be skeptical;

• Must be cautious;

• Must be ethical.

Common sources of bias

• Adjustment and narrowing

• Availability

• Selective perception

• Wishful thinking

• Base rate

• Cognitive conservatism

• Order effects

More sources of bias…

• Data saturation

• Law of small numbers

• Representativeness

• Desire for self-fulfilling prophecies

• Fact-value confusion

• Attribution errors

• Gambler’s fallacy

Even more sources of bias…

• Overconfidence

• Redundancy

• Motivational biases– Fear– Dishonesty– Greed

Avoiding bias

• Recognize that biases exist

• Sample information broadly

• Look for data on both sides

• Use models and consistency analysis

• Use proper scaling methods

• Give equal treatment to qualitative and quantitative data

Avoiding bias (cont’d)…

• Ensure that sample characteristics are associated and presented with the data

• Analyze past decisions

• Determine prior good and poor decisions

• Present information/data in several forms

Selection bias…

can occur in group comparison studies and retrospective studies

Observer bias…

Can occur in behavior observation research

Recall bias…

Can occur in survey research

-social desirability effects

Non-response bias…

Can occur in survey research

Research designs

• How was the sample selected?• Is the sample representative of population?• What type of study

– Descriptive?– Correlation?– Group comparison?– Qualitative or historical?

• What is known about the measures?• Data analyses appropriate?

Interpreting Research Results

• Are conclusions consistent with results?

• Practical vs. statistical significance?

• Can cause and effect be established?

• Importance of results for the population?