summary and conclusion - survey research and design in psychology
TRANSCRIPT
Survey Design II
Lecture 10
Survey Research & Design in Psychology
James Neill, 2017
Creative Commons Attribution 4.0
Summary & Conclusion
Image source: http://commons.wikimedia.org/wiki/File:Pair_of_merops_apiaster_feeding_cropped.jpg
7126/6667 Survey Research & Design in PsychologySemester 1, 2017, University of Canberra, ACT, AustraliaJames T. Neill
Description: Summary and conclusion to the Survey research and design in psychology lecture series.
Home page: http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychologyLecture page: http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Lectures/Summary_%26_conclusion
Image name: Pair of merops apiaster feeding cropped.jpgImage source: http://commons.wikimedia.org/wiki/File:Pair_of_merops_apiaster_feeding_cropped.jpgImage author: Pierre Dalous, http://commons.wikimedia.org/wiki/User:Kookaburra_81License: CC-by-SA, http://creativecommons.org/licenses/by-sa/3.0/deed.en
Overview
Module 1: Survey research and designSurvey research
Survey design
Module 2: Univariate and bivariateDescriptives & graphing
Correlation
Module 3: PsychometricsExploratory factor analysis
Psychometric instrument development
Module 4: Multiple linear regressionMLR I
MLR II
Module 5: Power & summaryPower & effect sizes
Summary and conclusion
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainLast chance to ask questions along the way as we review what the unit has coveredNo such thing as a silly question
Survey research
(Lecture 1)
Types of research
Surveys are used in all types of research: Experimental
Quasi-experimental
Non-experimental
What is a survey?
What is a survey?A standardised stimulus for converting fuzzy psychological phenomenon into hard data.
HistorySurvey research has developed into a popular social science research method since the 1920s.
Purposes of research
Purposes/goals of research:Information gatheringExploratory
Descriptive
Theory testing/buildingExplanatory
Predictive
Survey research
Survey researchPros include:Ecological validity
Cost efficiency
Can obtain lots of data
Cons include:Low compliance
Reliance on self-report
Survey design
(Lecture 2)
Self-administeredPros: cost
demand characteristics
access to representative sample
anonymity
Cons: non-response
adjustment to cultural differences, special needs
Survey types
Opposite for interview-administered surveys
Survey questions
Objective versus subjective questions:Objective - there is a verifiably true answer
Subjective - based on perspective of respondent
Open versus closed questions:Open - empty space for answer
Closed - pre-set response format options
Response formats
Dichotomous and Multichotomous
Multiple response
Verbal frequency scale (Never ... Often)
Ranking (in order Ordinal)
Likert scale (equal distances)
Graphical rating scale (e.g., line)
Semantic differential (opposing words)
Non-verbal (idiographic)
Level of measurement
Categorical/NominalArbitrary numerical labels
Could be in any order
OrdinalOrdered numerical labels
Intervals may not be equal
IntervalOrdered numerical labels
Equal intervals
RatioData are continuous
Meaningful 0
Sampling
Key terms(Target) population
Sampling frame
Sample
SamplingProbabilitySimple (random)
Systematic
Stratified
Non-probability
Convenience
Purposive
Snowball
Non-sampling biases
Acquiescence
Order effects
Fatigue effects
Demand characteristics
Hawthorne effect
Self-serving bias
Social desirability
Descriptives & graphing
(Lecture 3)
Getting to know data
Play with the data
Don't be afraid you can't break data
Check and screen the data
Explore the data
Get intimate with data
Describe the main features
Test hypotheses
Summary: LOM & statistics
If a normal distribution can be assumed, use parametric statistics (more powerful)
If not, use non-parametric statistics (less power, but less sensitive to violations of assumptions)
Descriptive statistics
What is the central tendency?Frequencies, Percentages (Non-para)
Mode, Median, Mean (Para)
What is the variability?Min, Max, Range, Quartiles (Non-para)
Standard Deviation, Variance (Para)
Normal distribution
MeanSD-ve Skew+ve SkewKurtRule of thumbSkewness and kurtosis in the range of -1 to +1 can be treated as approx. normal
Image source: Unknown
Skewness & central tendency
+vely skewed
mode < median < meanSymmetrical (normal)
mean = median = mode-vely skewed
mean < median < mode
Image source: Unknown
Principles of graphing
Clear purpose
Maximise clarity
Minimise clutter
Allow visual comparison
Univariate graphs
Bar graph
Pie chart
Histogram
Stem & leaf plot
Data plot /
Error bar
Box plot
Non-parametric
i.e., nominal or ordinal
}
}
Parametric
i.e., normally distributed interval or ratio
Non-normal parametric data can be recoded and treated as nominal or ordinal data.
Correlation
(Lecture 4)
Covariation
The world is made of covariations.
Covariations are the building blocks of more complex multivariate relationships.
Correlation is a standardised measure of the covariance (extent to which two phenomenon co-relate).
Correlation does not prove causation - may be opposite causality, bi-directional, or due to other variables.
Purpose of correlation
The underlying purpose of correlation is to help address the question:
What is therelationship or
association or
shared variance or
co-relation
between two variables?
What is correlation?
Standardised covariance
Ranges between -1 and +1, with more extreme values indicating stronger relationships
Correlation does not prove causation may be opposite causality, bi-directional, or due to other variables.
Types of correlation
Nominal by nominal:
Phi () / Cramers V, Chi-squared
Ordinal by ordinal:
Spearmans rank / Kendalls Tau b
Dichotomous by interval/ratio:
Point bi-serial rpb
Interval/ratio by interval/ratio:
Product-moment or Pearsons r
Correlation steps
Choose correlation and graph type based on levels of measurement.
Check graphs (e.g., scatterplot):
Linear or non-linear?
Outliers?
Homoscedasticity?
Range restriction?
Sub-samples to consider?
Correlation steps
Consider
Effect size (e.g., , Cramer's V, r, r2)
Direction
Inferential test (p)
Interpret/Discuss
Relate back to hypothesis
Size, direction, significance
Limitations e.g.,Heterogeneity (sub-samples)
Range restriction
Causality?
Interpreting correlation
Coefficient of determinationCorrelation squared
Indicates % of shared variance
Strength r r2 Weak: .1 - .3 1 10% Moderate: .3 - .5 10 - 25% Strong: > .5 > 25%
Assumptions & limitations
Levels of measurement
Normality
LinearityEffects of outliers
Non-linearity
Homoscedasticity
No range restriction
Homogenous samples
Correlation is not causation
Exploratory factor analysis
(Lecture 5)
What is factor analysis?
Factor analysis is a family of multivariate correlational data analysis methods for summarising clusters of covariance.
FA summarises correlations amongst items into a smaller number of underlying fuzzy constructs (called factors).
Steps / process
Test assumptions
Select extraction method and rotation
Determine no. of factors
(Eigen Values, Scree plot, % variance explained)
Select items
(check factor loadings to identify which items belong in which
factor; drop items one by one; repeat)
Name and define factors
Examine correlations amongst factors
Analyse internal reliability
Compute composite scores
Lecture
6
See also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Assumptions
Sample size 5+ cases per variables
(ideally 20+ cases per variable)
Another guideline: N > 200
Bivariate & multivariate outliers
Factorability of correlation matrix
(Measures of Sampling Adequacy)
Normality enhances the solution
Types of FA
PAF (Principal Axis Factoring):
Best for theoretical data exploration uses shared variance
PC (Principal Components):
Best for data reduction uses all variance
Rotation
Orthogonal (Varimax) perpendicular (uncorrelated) factors
Oblique (Oblimin) angled (correlated) factors
Consider trying both ways Are solutions different? Why?
How many factors to extract?Inspect EVs look for > 1 or sudden drop (inspect scree plot)
% of variance explained aim for 50 to 75%
Interpretability does each factor 'make sense'?
Theory does the model fit with theory?
Factor extraction
Item selection
An EFA of a good measurement instrument ideally has:a simple factor structure (each variable loads strongly (> +.50) on only one factor)
each factor has multiple loading variables (more loadings greater reliability)
target factor loadings are high (> .6) and cross-loadings are low (< .3), with few intermediate values (.3 to .6).
Psychometrics
instrument
development
(Lecture 6)
Psychometrics
Science of psychological measurement
Goal: Validly measure individual psychosocial differences
Develop and test psychological measures e.g., usingFactor analysis
Reliability and validity
Concepts & their measurement
Concepts name common elements
Hypotheses identify relations between concepts
Brainstorm indicators of a concept
Define the concept
Draft measurement items
Pre-test and pilot test
Examine psychometric properties
Redraft/refine and re-test
Measurement error
Deviation of measure from true score
Sources:Non-sampling (e.g., paradigm, respondent bias, researcher bias)
Sampling (e.g., non-representativeness)
How to minimise:Well-designed measures
Reduce demand effects
Representative sampling
Maximise response rate
Ensure administrative accuracy
Reliability
Consistency or reproducibility
TypesInternal consistency
Test-retest reliability
Rule of thumb> .6 OK
> .8 Very good
Internal consistencySplit-half
Odd-even
Cronbach's alpha
Validity
Extent to which a measure measures what it is intended to measure
Multifaceted
Compare with theory and expert opinion
Correlations with similar and dissimilar measures
Predicts future
Composite scores
Ways of creating composite (factor) scores: Unit weightingTotal of items or
Average of items
(recommended for lab report)
Regression weighting Each item is weighted by its importance to measuring the underlying factor (based on regression weights)
Writing up
instrument development
Introduction Review constructs & previous structures
Generate research question
Method Explain measures and their development
Results Factor analysis
Reliability of factors
Descriptive statistics for composite scores
Correlations between factors
Discussion Theory? / Measure? / Recommendations?
Multiple linear regression
(Lectures 7 & 8)
General steps
Develop model and hypotheses
Check assumptions
Choose type
Interpret output
Develop a regression equation
(if needed)
Linear regression
Best-fitting straight line for a scatterplot of two variables
Y = bX + a + ePredictor (X; IV)
Outcome (Y; DV)
Least squares criterion
Residuals are the vertical distance between actual and predicted values
MLR assumptions
Level of measurement
Sample size
Normality
Linearity
Homoscedasticity
Collinearity
Multivariate outliers
Residuals should be normally distributed
These residual slides are based on Francis (2007) MLR (Section 5.1.4) Practical Issues & Assumptions, pp. 126-127 and Allen and Bennett (2008)
Note that according to Francis, residual analysis can test:Additivity (i.e., no interactions b/w Ivs) (but this has been left out for the sake of simplicity)
Level of measurement and dummy coding
Levels of measurementDV = Continuous (Likert or ratio + normal)
IV = Continuous or dichotomous
Dummy codingConvert complex variable into series of dichotomous IVs
General steps
Develop model and hypotheses
Check assumptions
Choose type
Interpret output
Develop a regression equation
(if needed)
Multiple linear regression
Multiple IVs to predict a single DV:
Y = b1x1 + b2x2 +.....+ bixi + a + eOverall fit: R, R2, and Adjusted R2
CoefficientsRelation between each IV and the DV, adjusted for the other IVs
B, , t, p, and sr2
TypesStandard
Hierarchical
Stepwise / Forward / Backward
Summary:
Semi-partial correlation (sr)
In MLR, sr is labelled part in the regression coefficients table SPSS output
Square these values to obtain sr2, the unique % of DV variance explained by each IV
Discuss the extent to which the explained variance in the DV is due to unique or shared contributions of the IVs
Residual analysis
Residuals are the difference between predicted and observed Y values
MLR assumption is that residuals are normally distributed.
Examining residuals also helps assess: Linearity
Homoscedasticity
Interactions
In MLR, IVs may interact to: Increase the IVs' effect on the DV
Decrease the IVs' effect on the DV
Model interactions using hierarchical MLR: Step 1: Enter IVs
Step 2: Enter cross-product of IVs
Examine change in R2
Analysis of change
Analysis of changes over time can be assessed by: Standard
regression Calculate difference scores
(Post-score minus Pre-score) and use as a DV
Hierarchical MLR Step 1: Partial out baseline scores
Step 2: Enter other IVs to help predict variance in changes over time.
Writing up an MLR
Introduction Establish purpose
Describe model and hypotheses
Results Univariate descriptive statistics
Correlations
Type of MLR and assumptions
Regression coefficients
Discussion Summarise and interpret, with limitations
Implications and recommendations
Power & effect size
(Lecture 9)
Significance testing
Logic At what point do you reject H0?
History Started in 1920s & became very popular through 2nd half of 20th century
Criticisms Binary, dependent on N, ES, and critical
Practical significance Is an effect noticeable?
Is it valued?
How does it compare with benchmarks?
Inferential decision making
Statistical power
Power = probability of detecting a real effect as statistically significant
Increase by:
N
critical
ES
Power > .8 desirable
~ .6 is more typical
Can be calculated prospectively and retrospectively
Effect size
Standardised size of difference or strength of relationship
Inferential tests should be accompanied by ESs and CIs
Common bivariate ESs include: Cohens d
Correlation r
Cohens d - not in SPSS - use an online effect size calculator
Confidence interval
Gives range of certainty
Can be used for B, M, ES etc.
Can be examined Statistically (upper and lower limits)
Graphically (e.g., error-bar graphs)
Publication bias
Tendency for statistically significant studies to be published over non-significant studies
Indicated by gap in funnel plot file-drawer effect
Counteracting biases in scientific publishing:
low-power studies tend to underestimate real effects
bias towards publish sig. effects over non-sig. effects
Academic integrity
Violations of academic integrity are evident and prevalent amongst those with incentives to do so: Students
Researchers
Commercial sponsors
Adopt a balanced, critical approach, striving for objectivity and academic integrity
Unit outcomes
Learning outcomes
Design and conduct survey-based research in psychology;
Use SPSS to conduct and interpret data analysis using correlation-based statistics, including reliability, factor analysis and multiple regression analysis;
Communicate in writing the results of survey-based psychological research
Graduate attributes
Display initiative and drive, and use organisation skills to plan and manage workload
Employ up-to-date and relevant knowledge and skills
Take pride in professional and personal integrity
Use creativity, critical thinking, analysis and research skills to solve theoretical and real-world problems
Feedback
What worked well?
What could be improved?
Direct feedback
(e.g., email, discussion forum)
Interface Student Experience Questionnaire (ISEQ).
Open Office Impress
This presentation was made using Open Office Impress.
Free and open source software.
http://www.openoffice.org/product/impress.html
Column 1Column 2Column 3
Row 19.13.24.54
Row 22.48.89.65
Row 33.11.53.7
Row 44.39.026.2
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level