Using Multivariate Techniques for the Analysis of Survey Data:

A Case Example

Isabelle BourgeoisNatural Sciences and Engineering Research Council

University of Ottawa

Joint Conference of the CES and the AEAToronto, October 26th 2005

Rationale for Study

• Performance measurement activities in R&D often use complex techniques taken from the sciences and economics

• Methods are needed that capture more fully the non-economic benefits of research to support decision makers

• The use of multivariate analysis techniques taken from the social sciences can provide useful information based on survey results

Examples of Survey Measures

• Industry awareness of government research and researchers

• Satisfaction with past interactions between industry and government

• Level of trust in researchers and staff• Types of new processes or products

introduced to market

Use of Sophisticated Analytical Methods

• Traditional survey data analysis relies on descriptive statistics with little exploration of the relationships that exist between variables

• Harman (2004) example: Use of t-tests with little interpretation and no further research

Case Example

• Multivariate analysis conducted on data from the Postgraduate Scholarship (PGS) program managed by NSERC

• Data collection instrument: survey administered in the summer of 2005 (N=901)

• Attrition: 101• Response rate: 69.6% (n=557)• Survey results estimated to be accurate ±3%,

19 times out of 20

Variables

• Independent: gender, type of award received (master’s or doctoral), and main field of study

• Dependent: four seven-point scales on different aspects of the student’s experience with the PGS award

Preliminary Examination of Data

• Descriptive statistics to get a general sense of the survey results

• Verification of statistical assumptions most critical to multivariate analysis, normality and equality of the variance-covariance matrix (homoscedasticity)

Multivariate Respondent Profile

• MANOVA (Multivariate Analysis of Variance): yields information to answer questions about the combined effect of the dependent variables

• MANOVA conducted on each of the four scales using the gender, award type and field variables (Tables included in the paper)

Findings of MANOVA (1)

• Field variable found significant on its own in scale A2 and in combination with other dependent variables in scales A2 and C3

• Suggests that the field of study has an impact on a student’s undergraduate experiences (A2), as well as his or her graduate experiences (C3)

Findings of MANOVA (2)

• Award variable found significant on scale C2 which focused on relationship with supervisor and future career plans

• The award variable has an impact on the student’s relationship with supervisor and future career plans – Master’s and Ph.D. students have different experiences and expectations

Discriminant Function Analysis

• Close relative of the MANOVA• Instead of looking at the influence of the

independent variables on survey responses, it predicts group membership within each independent variable by using the responses to the scale items.

• Findings focus on items loading most highly for each variable – these provide the best discrimination between levels of the variable (examples)

Exploration of Scale Properties

• Aside from providing more sophisticated information about the results of the survey, multivariate analysis methods also provide information about the properties of the instrument itself.

Exploratory Factor Analysis

• Data summarization technique• Identifies underlying, or latent, dimensions in

a dataset that, when interpreted, describe the data in a smaller number of concepts

• The responses to the items on the PGS survey scales are hypothesized to vary as a function of one or several latent variables

Factors Extracted

• Nine meaningful factors extracted and retained for interpretation

• Example (Factor 1, Learning and Skill Development): Funding from NSERC will help me to complete my

degree faster Theoretical knowledge of the discipline Analytical techniques/experimental methods Use of laboratory equipment or instruments Project management Etc.

Factors Extracted1. Learning and Skill Development2. Supervision3. Resources4. Career Orientation5. No Debt6. Encouragement7. Collaborative Experiences8. Debt Load9. Job Prospects

Scale Reliability

• Reliability: Internal consistency of a scale – minimizes the contribution of random error to item scores

• Alpha coefficient is used to measure internal consistency and is interpreted as a correlation

• Alpha coefficients calculated for each of the scales and the combined items, as well as for the Factors extracted in the EFA to see which model fits best

Reliability Comparison

Original Scales

Scale A2: 0.59Scale C1: 0.81Scale C2: 0.77Scale C3: 0.85

Combined Items: 0.82

Factors

Factor 1: 0.84Factor 2: 0.91Factor 3: 0.81Factor 4: 0.64Factor 6: 0.50Factor 8: 0.42Factor 9: 0.24

Advantages of Multivariate Analysis

• Easy to do using standard statistical software packages

• Interpretation is fairly straightforward• Adds to descriptive data by highlighting the

influence of the combined dependent variables on the survey responses

• Allows for a review of the instrument properties and provides suggestions for improvement