company logo types of research jan Štochl. content 2. types of research designs 3. experimental...

Company

LOGO

Types of researchTypes of research

Jan Štochl

ContentContent

2. Types of research designs 2. Types of research designs

3. Experimental design 3. Experimental design

4. Quasi-experimental design 4. Quasi-experimental design

5. Nonexperimental design 5. Nonexperimental design

1. Introduction to research 1. Introduction to research

What is research?What is research?

Research is an active, diligent and systematic process of inquiry in order to discover, interpret or revise facts, events, behaviours, or theories, or to make practical applications with the help of such facts, laws or theories.

Phases of researchPhases of research

Phase 1Phase 1 Phase 2Phase 2 Phase 3Phase 3

1. Problem 2. Studiing recent publications3. Decision on the problem4. Determine design

and methods5. Organisation

1. Taking a sample from population

2. Pilot study3. Correction of

errors4. Data gathering

1. Data analysis2. Interpretation of

results3. Discussion4. Publication5. (Crossvalidation)

Random assignmentRandom assignment

• The assignment of individuals in the pool of all potential participants to either the experimental (treatment) group or the control group in such a manner that their assignment to a group is determined entirely by chance

• It is an attempt to try to minimize effects of random variables by distributing them randomly across groups

• Individuals are placed into groups or treatment conditions in such a way that each person has an equal chance of being selected for any group or treatment. In addition, placement of any individual into a group or treatment condition does not influence the placement of any other person

Types of researchTypes of research

What is Research Design? What is Research Design?

• Research design can be thought of as the structure of research -- it is the "glue" that holds all of the elements in a research project together

• Design is described using a concise notation that enables us to summarize a complex design structure efficiently

• the "elements" that a design includes:– Observations or Measures (O)

– Treatments or Programs (X)

– Groups (lines)

– Assignment to Group (R = random assignment; N = nonequivalent groups; C = assignment by cutoff)

– Time (goes from left to right)

Examples of research designExamples of research design

Examples of experiment, quasi-experiment and nonexperiment

Examples of experiment, quasi-experiment and nonexperiment

Can you „read“ it?

Experimental designExperimental design

• Experimental designs are often touted as the most "rigorous" of all research designs or, as the "gold standard" against which all other designs are judged

• Experiment is probably the strongest design with respect to internal validity

• Can „prove“ cause-effect relationship

Establishing cause-effectEstablishing cause-effect

• Three major conditions have to be met:

1. Temporal Precedence

2. Covariation of the Cause and Effect

3. No Plausible Alternative Explanations

Experimental designExperimental design

• simultaneously address the two following propositions:

If X, then Y and

If not X, then not Y

• Or, once again more colloquially:

If the program is given, then the outcome occurs and

If the program is not given, then the outcome does not occur

Mill´s methodsMill´s methods

1. The Joint Method of Agreement and Difference

2. The Method of Agreement

3. The Method of Difference

4. The Method of Residues

5. The Method of Concomitant Variations

Classifying experimental designClassifying experimental design

• Signal enhancers

• Noise reducers

Signal-enhancing experimental designs Signal-enhancing experimental designs

• Also called factorial designs

Possible outcomes of factorial designsPossible outcomes of factorial designs

Possible outcomes of factorial designsPossible outcomes of factorial designs

Possible outcomes of factorial designsPossible outcomes of factorial designs

Possible outcomes of factorial designsPossible outcomes of factorial designs

Possible outcomes of factorial designsPossible outcomes of factorial designs

Possible outcomes of factorial designsPossible outcomes of factorial designs

Possible outcomes of factorial designsPossible outcomes of factorial designs

Statistical analysis of factorial experimental designsStatistical analysis of factorial experimental designs

Features of factorial experimental designFeatures of factorial experimental design

• Great flexibility for exploring or enhancing the “signal” (treatment) in our studies

• Efficiency

• The only effective way to examine interaction effects.

Noise-reducing experimental designs Noise-reducing experimental designs

• Two major types:

1. covariance designs

2. blocking designs

Covariance designsCovariance designs

• Also called analysis of covariance design (ANCOVA)

• Example: simple randomized two group pretest postest design

• measure is not necessarily the same

Basic idea behind noise reductionBasic idea behind noise reduction

Statistical analysis of covariance designsStatistical analysis of covariance designs

• The so-called ANCOVA

Summary of covariance designsSummary of covariance designs

• It "adjusts" posttest scores for variability on the covariate (pretest). This is what we mean by "adjusting" for the effects of one variable on another

• any continuous variable can be used as a covariate, but the pretest is usually best. (Because the pretest is usually the variable that would be most highly correlated with the posttest)

• Because it's so highly correlated, when you "subtract it out" or "remove' it, you're removing more extraneous variability from the posttest.

Randomized block designsRandomized block designs• It is a research design's equivalent to stratified random sampling

• Useful if some homogeneous groups are recognised in the sample

• Researcher divide the sample into relatively homogeneous subgroups or blocks

• The key idea is that the variability within each block is less than the variability of the entire sample.

• Each estimate of the treatment effect within a block is more efficient than estimates across the entire sample

• When we pool these more efficient estimates across blocks, we should get an overall more efficient estimate than we would without blocking.

Example of randomized block designExample of randomized block design

Statistical analysis of randomized block designs Statistical analysis of randomized block designs

• Regression analysis

Hybrid experimental designsHybrid experimental designs• New strains that are formed by combining features

of more established designs

• The Solomon Four-Group Design

• Switching Replications Design

The Solomon Four-Group DesignThe Solomon Four-Group Design

• Deals with a potential testing threat.

• A testing threat occurs when the act of taking a test affects how people score on a retest or posttest.

The Solomon Four-Group Design – possible outcomesThe Solomon Four-Group Design – possible outcomes

Switching replicationsSwitching replications

• One of the strongest experimental design

Switching replications – possible outcomesSwitching replications – possible outcomes

Two-group experimental designTwo-group experimental design

• The simplest of all experimental designs is the two-group posttest-only randomized experiment

• Typically we measure the groups on one or more measures and we compare them by testing for the differences between the means using a t-test or one way Analysis of Variance (ANOVA).

Two-group experimental designTwo-group experimental design

• The simplest of all experimental designs

• The posttest-only randomized experiment is strong against the single-group threats to internal validity because it's not a single group design!

• It's strong against the all of the multiple-group threats except for selection-mortality

• It is susceptible to all of the social interaction threats to internal validity

Statistical analysis of two group designStatistical analysis of two group design

• The so-called t-test – assessing the mean difference

Quasi-experimental designsQuasi-experimental designs

• Looks like an experimental design but lacks the key ingredient -- random assignment

• With respect to internal validity, they often appear to be inferior to randomized experiments

• Nonequivalent qroups design and regression-discontinuity design

Nonequivalent groups designNonequivalent groups design

• Most frequent in the social research

• Identical to the Analysis of Covariance design except that the groups are not created through random assignment

• This nonrandom assignment complicates the statistical analysis a lot

• susceptible to the internal validity threat of selection

The regression-discontinuity design The regression-discontinuity design

• C indicates that groups are assigned by means of a cutoff score

• All persons on one side of the cutoff are assigned to one group; all persons on the other side of the cutoff are assigned to the other

• Need a continuous quantitative pre-program measure

Logic of regression-discontinuity designLogic of regression-discontinuity design

Selection of cutoff and interpretation of resultsSelection of cutoff and interpretation of results

Other quasi-experimental designsOther quasi-experimental designs

• The Proxy Pretest Design

• The Separate Pre-Post Samples Design

• The Double Pretest Design

• The Switching Replications Design

• The Nonequivalent Dependent Variables (NEDV) Design

• The Regression Point Displacement (RPD) Design

Relationships between pre-post designsRelationships between pre-post designs

Minimizing threats to validityMinimizing threats to validity

• By Argument

• By Measurement or Observation

• By Design

• By Analysis

• By Preventive Action

Design Construction - Basic Design Elements Design Construction - Basic Design Elements

1. Time

2. Program(s) or Treatment(s)

3. Observation(s) or Measure(s)

4. Groups or Individuals

Advances in Quasi-ExperimentationAdvances in Quasi-Experimentation

• The Role of Judgment• The Case for Tailored Designs• The Crucial Role of Theory• Attention to Program Implementation• The Importance of Quality Control• The Advantages of Multiple Perspectives• Evolution of the Concept of Validity• Development of Increasingly Complex Realistic

Analytic Models

Nonexperimental designNonexperimental design

Observation

Questionnaire

Interview

Rating scale

Correlation study

Nonexperimental design

Nonexperimental design

Cannot prove causatio

n!!!

Survey researchSurvey research

• Involves any measurement procedures that involve asking questions of respondents

• Can be anything from short paper-and-pencil feedback form to an intensive one-on-one in-depth interview

• Usually surveys are divided into two broad categories: interviews and questionnaires

Types of questionnairesTypes of questionnaires

• mail survey

• group administered questionnaire

• household drop-off survey

Types of interviewTypes of interview

• personal interview

• telephone interview

Selecting the survey method - population issuesSelecting the survey method - population issues

• What follows is the set of considerations with population and its accesibility:

• Can the population be enumerated? • Is the population literate? • Are there language issues? • Will the population cooperate? • What are the geographic restrictions?

Selecting the survey method - sampling issuesSelecting the survey method - sampling issues

• What data is available?

• Can respondents be found?

• Who is the respondent?

• Can all members of population be sampled?

• Are response rates likely to be a problem?

Selecting the survey method - question issuesSelecting the survey method - question issues

• What types of questions can be asked?

• How complex will the questions be?

• Will screening questions be needed?

• Can question sequence be controlled?

• Will lengthy questions be asked?

• Will long response scales be used?

Selecting the survey method - content issuesSelecting the survey method - content issues

• Can the respondents be expected to know about the issue?

• Will respondent need to consult records?

Selecting the survey method - bias issuesSelecting the survey method - bias issues

• Can social desirability be avoided?

• Can interviewer distortion and subversion be controlled?

• Can false respondents be avoided?

Selecting the survey method - administrative issuesSelecting the survey method - administrative issues

• Costs

• Facilities

• Time

• Personnel

Constructing the surveyConstructing the survey

• Survey questions can be divided into two broad types: structured and unstructured.

• From an instrument design point of view, the structured questions pose the greater difficulties

• From a content perspective, it may actually be more difficult to write good unstructured questions

Types of questionsTypes of questions

• Dichotomolus questions

Types of questionsTypes of questions

• Questions based on level of measurement

• Purely nominal

• Ordinal like

Types of questionsTypes of questions

• Likert response scale

• Semantic differential

Types of questionsTypes of questions

• Cumulative or Guttman scale

Types of questionsTypes of questions

• Filter or Contingency Questions

InterviewsInterviews

• Another type of survey design

• From the psychological point of view the most difficult method

• Usually personal or phone

The role of interviewerThe role of interviewer

• Locate and enlist cooperation of respondents

• Motivate respondents to do good job

• Observe quality of responses

• Conduct a good interview

Training the interviewerTraining the interviewer

• major topics that should be included in interviewer training:

1) State who is sponsor of research

2) Describe the entire study

3) Teach enough about survey research

4) Explain interviewer bias

5) ….

Interviewer kitInterviewer kit

• A "professional-looking" 3-ring notebook (this might even have the logo of the company or organization conducting the interviews)

• Maps• Sufficient copies of the survey instrument • Ifficial identification (preferable a picture ID)• A cover letter from the Principal Investigator

or Sponsor • A phone number the respondent can call to

verify the interviewer's authenticity

Pluses and minuses of survey researchPluses and minuses of survey research

Qualitative researchQualitative research

• The purpose of this section is to introduce you to the idea of qualitative research (and how it is related to quantitative research) and give you some orientation to the major types of qualitative research data, approaches and methods.

Qualitative research considerationsQualitative research considerations

• Do you want to generate new theories or hypotheses?

• Do you need to achieve a deep understanding of the issues?

• Are you willing to trade detail for generalizability?

The Qualitative-Quantitative Debate The Qualitative-Quantitative Debate

• Quantitative research excels at summarizing large amounts of data and reaching generalizations based on statistical projections. 

• Qualitative research excels at "telling the story" from the participant's viewpoint, providing the rich descriptive detail that sets quantitative results into their human context.

• Necessity of mixed approach

Qualitative and Quantitative Data Qualitative and Quantitative Data

• All qualitative data can be coded quantitatively

• All quantitative data is based on qualitative judgment

Qualitative and Quantitative AssumptionsQualitative and Quantitative Assumptions

Many people think that:

Quantitative research is confirmatory and deductive in nature. Qualitative research is exploratory and inductive

in nature.

• This can be however misleading

Qualitative and Quantitative AssumptionsQualitative and Quantitative Assumptions

• Differences are based more on epistemological assumptions, i.e. many qualitative researchers believe that the best way to understand any phenomenon is to view it in its context.

• Many qualitative researchers also operate under different ontological assumptions about the world.  They don't assume that there is a single unitary reality apart from our perceptions.

Qualitative dataQualitative data

• In-Depth Interviews – includes both individual interviews (e.g., one-on-one) as well as "group" interviews (including focus groups).

• Direct Observation – it is meant in a very broad sense. Differs from interviewing in that the observer does not actively query the respondent.

• Written Documents - It can include newspapers, magazines, books, websites, memos, transcripts of conversations, annual reports, and so on.

Qualitative approachesQualitative approaches

• Ethnography-studying an entire culture

• Phenomenology-people's subjective experiences and interpretations of the world

• Field Research-the researcher goes "into the field" to observe the phenomenon in its natural state

• Grounded Theory

Grounded TheoryGrounded Theory

• developed by Glaser and Strauss in the 1960s

• The purpose of grounded theory is to develop theory about phenomena of interest

•

Grounded Theory – key strategiesGrounded Theory – key strategies

• Coding is a process for both categorizing qualitative data and for describing the implications and details of these categories.

• Memoing is a process for recording the thoughts and ideas of the researcher as they evolve throughout the study

• Integrative diagrams and sessions are used to pull all of the detail together, to help make sense of the data with respect to the emerging theory.

Qualitative MethodsQualitative Methods

• Participant Observation

• Direct Observation

• Unstructured Interviewing

• Case Studies

Random selectionRandom selection

• Probability sampling method is any method of sampling that utilizes some form of random selection

• It is necessary to set up some process or procedure that assures that the different units in your population have equal probabilities of being chosen.

• Past - picking a name out of a hat, or choosing the short straw

• Present – computers and random numbers

Random selection versus random assignmentRandom selection versus random assignment

• Random selection is how you draw the sample of people for your study from a population.

• Random assignment is how you assign the sample that you draw to different groups or treatments in your study

Simple random samplingSimple random sampling

• Objective: To select n units out of N such that each NCn has an equal chance of being selected

• Procedure: Use a table of random numbers, a computer random number generator, or a mechanical device to select

the sample.

Stratified Random SamplingStratified Random Sampling

• Objective: also sometimes called proportional or quota random sampling, involves dividing your population into homogeneous subgroups and then taking a simple random sample in each subgroup

• Procedure: Divide the population into non-overlapping groups (i.e., strata) N1, N2, N3, ... Ni, such that N1 + N2 + N3 + ... + Ni = N. Then do a simple random sample of f = n/N in each strata

Systematic Random Sampling Systematic Random Sampling

• Procedure:Number the units in the population from 1 to N; decide on the n (sample size) that you want or need;k = N/n = the interval size;randomly select an integer between 1 to k then take every kth unit

Cluster (Area) Random Sampling Cluster (Area) Random Sampling

• Objective: This strategy will help us to economize on our

mileage • Procedure: Divide population into clusters (usually along

geographic boundaries); randomly sample clusters; measure all units within sampled clusters

Multi-Stage Sampling Multi-Stage Sampling

• Various combinations of previously mentioned sampling methods (simple, stratified, systematic and cluster)

Block DiagramBlock Diagram

TEXT TEXT TEXT TEXT

TEXT TEXT TEXT TEXT

TableTable

TEXT TEXT TEXT TEXT TEXT

Title A

Title B

Title C

Title D

Title E

Title F

3-D Pie Chart3-D Pie Chart

TEXT

TEXT

TEXT

TEXTTEXT

TEXT

Marketing DiagramMarketing Diagram

Title

TEXT TEXTTEXT TEXT