1

Student Surveys and the Student Learning Experience

Michael ProsserHigher Education Academy

2

Academy Mission

The Academy’s mission is to help institutions, discipline groups and all staff to provide the best possible learning experience for their students.

3

Presentation based upon:

• Over 20 years research into the student learning experience in higher education in the United Kingdom, Australia, Sweden and Hong Kong

• Experience in Australia with the Course Experience (Perceptions) Questionnaire

The interpretation of the results of student surveys is not value or theory free:

• Interpretations in terms of student satisfaction and rankings

• Interpretations in terms of student perceptions relating to student learning

4

  Overview of the student learning perspective

Figure 1: Model of Student Learning

CHARACTERISTICS OF THE STUDENT (e.g. previous experiences, current understanding) STUDENTS'

PERCEPTIONS OF CONTEXT (e.g. good teaching, clear goals)

COURSE AND DEPARTMENTAL LEARNING CONTEXT (e.g. course design, teaching methods, assessment)

STUDENTS' APPROACHES TO LEARNING (how they learn e.g. surface/deep)

STUDENTS' LEARNING OUTCOMES (what they learn quantity/quality)

5

STUDENT APPROACHES TO LEARNING

 

Surface Approach

Intention to reproduce

- rote memorise information needed for assessment

- failure to distinguish principles from examples

- treat tasks as external impositions

- focus on discrete elements without integration

 

Deep Approach

Intention to understand

- meaningfully memorise information for later use

- relate new ideas to previous knowledge

- relate concepts to everyday experiences

- relate evidence to conclusions 

6

Approach to learning represents the relationship between the student and the task or subject

Approaches vary between subjects

Approaches vary between tasks within subjects

Approaches not stable but variable

Surface and deep approaches describe a fundamental variation in the way students approach learning – they do not describe the rich detail in the way students approach their learning

7

STUDENTS’ PERCEPTIONS OF THE LEARNING CONTEXT

Research by Entwistle and Ramsden (1983) using interviews and questionnaires identified a number of student perceptions relating to the way they approached their studies

Student perceptions of:

Quality of teaching – including quality of feedback

(NSS: Teaching, Assessment and Feedback, Academic Support))

Clearness of goals of course and standards of assessment

(NSS: Assessment and feedback)

Workload so high that it was not possible to understand everything

Assessment measuring reproduction and not understanding

were found to relate to how they approach their studies and to learning outcomes (exam results and other indicators)

8

RELATIONSHIP BETWEEN PERCEPTIONS AND APPROACHES

Study of 8837 students in 51 first year subjects around Australia

Amongst the data collected were students responses to:

contextualised Ramsden's Course Experience Questionnaire and a

contextualised Biggs Study Process Questionnaire

1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

9

Approaches to Study

Surface Approach

32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture

35. The best way for me to understand what technical terms mean is to remember the textbook definitions

 

Deep Approach

28. I try to relate ideas in this subject to those in other subjects, wherever possible

34. In trying to understand new ideas, I often try to relate them to real life situations to which they might apply.

 

Biggs Study Process Questionnaire

10

Student Perceptions Of Learning Context

Good Teaching

15. The staff made a real effort to understand difficulties students might be having with their work.

Clear Goals and Standards

1. It was always easy to know the standard of work expected

6. I usually had a clear idea of where I was going and what was expected of me in this subject.

Appropriate Workload

25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended (-).

Appropriate Assessment

8. To do well in this subject, all you really need is a good memory (-).

Ramsden’s Course Experience Questionnaire

11

Analysis conducted using:

1. Individual student as unit of analysis across all first year subjects

2. Individual student as unit of analysis within first year subjects subjects

3. First year subjects as unit of analysis

12

Factor Analysis of Perceptions of T & L Context and Approach to Study - individual

_________________________________________________________________

Scale Factors

____________________________

1 2

_________________________________________________________________

Perceptions of Context

Good teaching .80

Clear Goals and Standards .67

Appropriate Workload -.69

Appropriate Assessment -.65

Approach to Study

Surface Approach .81

Deep Approach .73

_________________________________________________________________

Principal Components, Varimax Rotation, n=8837

13

 At individual student level:

 

1.   A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback)

 

2.   A surface approach is associated with perceptions that the workload is too high and assessment tests reproduction

That is, variation in individual students’ perceptions of the learning context across all subjects is associated with the approaches to study (and learning outcome)

1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

14

Factor Analysis of Perceptions of T & L Context and Approach to Study – within first year medical subjects in different universities_________________________________________________________________

Scale Subject A Subject B Subject C

________ ________ ________

1 2 1 2 1 2

_________________________________________________________________

Perceptions of Context

Good teaching .84 .77 .79

Clear Goals and Standards .68 .70 .69

Appropriate Workload -.73 -.68 -.69

Appropriate Assessment -.70 -.60 -.60

Approach to Study

Surface Approach .75 .84 .80

Deep Approach .62 .64 .72

__________________________________________________________________

15

 Within each subject:

 

1.   A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback)

 

2.   A surface approach is associated with perceptions that the workload is too high and assessment tests reproduction

That is, variation in students’ perceptions of the learning context within subjects is associated with the approaches to study within subjects – within subject variation in perception is not measurement error.

1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

16

Factor Analysis of Perceptions of T & L Context and Approach to Study - subject

_________________________________________________________________

Scale Factors

__________________________

1 2

_________________________________________________________________

Perceptions of Context

Good teaching .81 -.34

Clear Goals and Standards .78

Appropriate Workload -.75

Appropriate Assessment -.87

Approach to Study

Surface Approach .91

Deep Approach .82

_________________________________________________________________

Principal Components, Variamax Rotation, n=51

17

 At subject level:

 

1.   A deep approach is associated with perceptions that the teaching is good and the goals and standards are clear (NSS: Teaching, Assessment and Feedback)

 

2.   A surface approach is associated with perceptions that the teaching is not so good, workload is too high and assessment tests reproduction

That is, variation in students’ perceptions of the learning context between subjects is associated with the approaches to study

1994-1996: Australian Research Council ; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS

18

Example for student learning in first year mathematics

In a study of first year university mathematics students in a large research intensive university, analyses of short open ended written statements by 236 students identified:

Conceptions of Mathematics:

Fragmented Conceptions

A. Maths as numbers, rules and formulae

B. Maths as numbers etc with applications to problems

Cohesive Conceptions

C. Maths as a way of thinking

D. Maths as a way of thinking for complex problem solving

E. Maths provides insights for understanding the world

19

Fragmented Conceptions

A. Maths as numbers, rules and formulae

Maths is the study of numbers, and the application of various methods to changing numbers

B. Maths as numbers etc with applications to problems

Maths is the study of numbers and their applications in other subjects and the physical world

20

Cohesive Conceptions

C. Maths as a way of thinking

Maths is the study of logic. Numbers and symbols are used to study life in a systematic perspective and requires the mind to think in a logical and often precise manner

D. Maths as a way of thinking for complex problem solving

Maths is an abstract reasoning process which can be utilized to explore and solve problems

E. Maths provides insights for understanding the world

Techniques for thinking about observable, physical phenomena in a quantitative way and also for thinking more abstractly with little or no relation to the directly observable universe.

21

Approaches to Studying Mathematics:

Surface approach – focus on reproducing

A. Learning by rote memorisation – to reproduce knowledge and proceduresB. Learning by doing lots of examples – to reproduce knowledge and

procedures

Deep approach – focus on understanding

C. Learning by doing lots of examples – to understand theory and conceptsD. Learning by doing difficult problems – to understand and to relate to other

knowledgeE. Learning by doing difficult problems and studying theory to look for

situations that theory may apply.

22

Approaches to Studying Mathematics:

Surface approach – focus on reproducing

A. Learning by rote memorisation – to reproduce knowledge and procedures

I liked calculus because I could remember formulas which is how I used to study. I would rote learn all the formulas and summarize all my theoretical notes

B. Learning by doing lots of examples – to reproduce knowledge and procedures

The way I go about studying for mathematics is by doing lots of examples and questions. Firstly I would study the notes and learn the formulas, then I put all of that to use by doing heaps of examples

23

Deep approach – focus on understanding

D. Learning by doing difficult problems – to understand and to relate to other knowledge

After listening to explanation of how a particular maths works the most essential features a repetition to develop speed (this usually consists of boring menial tasks) and an equal component of very difficult problems which require a great deal of thought to explore that area and its various properties and their consequences

E. Learning by doing difficult problems and studying theory to look for situations that theory may apply.

Read the relevant theory and try to get on the same “wavelength” as the person who actually discovered it. Before I attempt any problems I try to think where you can use the concept: i.e. what the concept was invented for. Then I attempt problems (on my own).

24

Relationship between Conception of Mathematics and Approach to Studying Mathematics ____________________________________________________________Conception Approach Total

_________________Surface Deep

____________________________________________________________Fragmented (A & B) 179 17 196

Cohesive (C, D, E) 4 36 40  Total 183 53 236____________________________________________________________Chi-square=126, p<.001Phi=.57ES=2.1

Crawford, K., Gordon, S., Nicholas, J. and Prosser, M. (1998) Qualitatively different experiences of learning mathematics at university. Learning and Instruction, 8, 455-468.

25

Defining Items from the Conceptions of Mathematics Questionnaire

Scale Item Item

Fragmented Conception

Cohesive Conception

Mathematics is figuring out problems involving numbers

Mathematics is a logical system which helps explain the things around us

26

Cluster Analysis of Prior Experiences and Understandings, Perceptions and Approaches and Post Experiences and Understandings Subscales

27

___________________________________________________________________________Variables Mean Surface Mean Deep

(n=147) (n=127)___________________________________________________________________Prior Experiences and Understandings Prior Fragmented Conception of Mathematics 0.31 -0.46 Prior Cohesive Conception of Mathematics -0.39 0.45 Prior Surface Orientation 0.34 -0.44 Prior Deep Orientation -0.56 0.59 Prior Academic Ranking (TER) -0.29 0.35Perceptions and Approaches Good Teaching -0.37 0.37 Clear Goals and Standards -0.23 0.24 Appropriate Workload -0.30 0.32 Appropriate Assessment -0.12 0.12 Surface Approach 0.44 -0.45 Deep Approach -0.54 0.57Post Experiences and Understanding Post Fragmented Conception 0.34 -0.41 Post Cohesive Conception -0.27 0.27 Achievement (Final Mark in Mathematics) -0.34 0.40___________________________________________________________________

28

Conclusions

The Table shows clear relations between prior experiences, perceptions of teaching and approaches and post experience and understanding.

The analysis identified 2 groups of students

In particular, those students:

• with more fragmented and less cohesive conceptions on entry, describe having adopted more of a surface approach to studying at high school and less of a deep approach,

• perceive the teaching to be poorer, to adopt more of a surface and less of a deep approach to studying at university, and

• have more fragmented and less cohesive conceptions after studying and poorer achievement

The opposite was found for the other group

29

  Overview of the student learning perspective

Figure 1: Model of Student Learning

CHARACTERISTICS OF THE STUDENT (e.g. previous experiences, current understanding) STUDENTS'

PERCEPTIONS OF CONTEXT (e.g. good teaching, clear goals)

COURSE AND DEPARTMENTAL LEARNING CONTEXT (e.g. course design, teaching methods, assessment)

STUDENTS' APPROACHES TO LEARNING (how they learn e.g. surface/deep)

STUDENTS' LEARNING OUTCOMES (what they learn quantity/quality)

30

Substantial amount of research linking results on surveys such as the NSS with student approaches to study and learning outcomes

Student perceptions are a function of both their prior experiences and understandings and the course design and teaching

Do not improve student satisfaction by focusing on satisfaction - the individual items or scales – need to better understand why they responded the way they

Spread of results – proportions responding in certain ways – better than mean of responses

Substantial disciplinary variation – unreasonable to compare between disciplines

Can only expect small effect sizes in changes in scores over time - .2 of a SD – if over 3 - 5 years changes of the order of .1 to .2 points (3.5 to 3.6 or 3.7).

31

• Focus on the student perception rather than satisfaction – better understand their perceptions – why they respond the way they have

• Comprehensive and aligned set of student evaluation instrument designed and interpreted in terms of student perceptions

• Institutions quality assurance and academic development and CPD activities aligned with the underlying model

• Treat data as indicators requiring further investigation – focus groups, open ended responses etc

32

Academy Response

1. Conferences relating to student experiences

2. Workshops and seminars to interested HEI’S and groups of staff

3. Work with interested institutions on ways to improve their students’ experiences as indicated by the NSS

4. Suggesting ways of further developing and / or improving future surveys

1. Review of dissemination site

2. Development of a bank of optional items, a selection of which individual institutions may wish to add to their NSS survey

33

Variable Variable

____________________________________________________________________

T A S O R D OS

Teaching 1 .39 .66 .45 -.01 .50 .80

Assessment 1 .55 .34 -.06 .32 .37

Support 1 .62 .21 .66 .77

Organisation 1 .14 .31 .77

Resources 1 .04 .17

Development 1 .55

OS 1

Correlation matrix of NSS – institutional