new approaches to course and subject improvement through ...aair.org.au/app/webroot/media/pdf/aair...

New approaches to course and subject improvement through qualitative analysis of student comments in evaluation surveys

Carolyn Newbigin BA, B Psych, Grad Dip Psych, Grad Cert Social Research ADSRI

Social Research Specialist Planning & Quality Unit

Workshop Session: Details and Agenda 1:50 pm – 3:00 pm (70 minutes) Monday 11th November, 2012

Time Duration Topic

1:50 pm – 1:55 pm 5 minutes Introduction to the Workshop • Details of the session • Run through the agenda

1:55 pm – 2:05 pm 10 minutes Getting to know the participants Group discussion: Where is everyone from, roles in other institutions?

2:05 pm – 2:25 pm 20 minutes Summative Content Analysis methodology at UTS • Quantifying the qualitative • What is included in a report to a faculty • NVivo coding of student comments • Visualising qualitative data using word clouds Group discussion: Q&A on the workshop so far

2:25 pm – 2:35 pm 10 minutes Common themes which emerge from student comments at UTS • Positive Aspects • ‘Ideas for Improvement’ Group discussion: Are these similar to the common themes found in your institutions?

2:35 pm – 2:45 pm 10 minutes Important issues when dealing with student feedback Group discussion: Scenario cards

2:45 pm – 2:55 pm 10 minutes Overview of how student comments are used to improve learning and teaching at UTS Group discussion: Reflections on the UTS approach, similarities and differences to approach taken at other institutions

2:55 pm – 3:00 pm 5 minutes Rate this session

Getting to know the participants

UTS:PLANNING AND QUALITY UNIT

November 2012

Go to page 2 of the workbook

1:55 pm – 2:05 pm 10 minutes Getting to know the participants

Group discussion: Where is everyone from, roles in other institutions?

Summative Content Analysis methodology at UTS


November 2012

2:05 pm – 2:25 pm 20 minutes Summative Content Analysis methodology at UTS • Quantifying the qualitative

• What is included in a report to a faculty

• NVivo coding of student comments

• Visualising qualitative data using word clouds

Group discussion: Q&A on the workshop so far

Go to page 3 and 4 of the workbook

Quantifying the Qualitative


November 2012

Why is it important?

Lecturers and heads of school paying too much attention to individual

comments:

• isolated incidents;

• personality clash;

• unfair appraisal detrimentally affecting a lecturer’s career;

• changes being made to the curriculum which aren’t based on pervasive

issues which are important to students.

What can be done?

Measuring and reporting confidence levels that the sample size is adequate,

i.e., the responses you have are representative of the entire class experience.

http://www.raosoft.com/samplesize.html


November 2012


November 2012

Confidence Levels A ‘Confidence level’ is a term which refers to the level of certainty that the responses reported would be indicative of the entire group of graduates. This is calculated using the number of graduates (population size), the number of responses (sample size) and other standard margin of error terms in an algorithm. An online tool including further details about how confidence levels are calculated is available at http://www.raosoft.com/samplesize.html. As would be expected, the greater the proportion of the graduates who respond to the questions, the higher the confidence that the responses will be representative of the true class experience. Regardless of the confidence levels reported, the responses are the legitimate experiences of individual students. However, it should be kept in mind that when the confidence level is LOW the typical student experience may have varied somewhat compared with the responses provided. There are three categories of confidence level used in this report:

HIGH 75% and above (95% and above is optimal)

MODERATE 60-75%

LOW Less than 60%

Quantifying the Qualitative


November 2012

What else is included in a report to a faculty?

Important Note This report contains data based on CEQ results from evaluation surveys completed by former UTS students, and includes the names of some staff members. Following the delivery of this report by PQU to the Faculty of *****************, access to this

report by faculty staff should be determined in accordance with the policy on the use of teaching evaluations (as set out in the Academic Staff Agreement).

This report and its contents are not intended for distribution outside UTS.


November 2012


Year of Survey Pass rate Number of

graduates who

completed the

course

Number of graduates

who completed the

quantitative questions

Number of graduates

who completed the

qualitative questions

2009 81.2% 243 students 148 students

(60.9% of total)

123 students

(50.6% of total)


(60.0% of total)

129 students

(51.6% of total)


(72.9% of total)

180 students

(62.5% of total)

Total Av.

82.3%

781 students 508 students

(65.0% of total)

432 students

(55.3% of total)

Confidence level that the overall sample size

is adequate

For qualitative comments on the SFS

VERY HIGH (optimal)

99.8% confidence

Confidence level that the overall sample size

is adequate

For quantitative scores on the SFS

VERY HIGH (optimal)

99.9% confidence


November 2012


TABLE 1.02

Quantitative Summary Table – (Name of

Subject here)

2009

1st Semester

(N=16)

LOW confidence

2010

1st Semester

(N=23)

LOW confidence

2011

1st Semester

(N=5)

LOW confidence

WEIGHTED

AVERAGE FROM

2009-2011

(N=44)

MODERATE

confidence

Average Satisfaction with Subject 4.46 4.04 4.78 4.28

The subject was delivered in a way which was

consistent with its stated objectives. 4.38 4.04 4.60 4.23

My learning experiences in this subject were

interesting and thought provoking. 4.57 3.96 4.80 4.27

I found the assessment fair and reasonable. 4.38 3.90 4.60 4.15

There were appropriate resources available to

support the subject. 4.57 4.00 4.80 4.30

I received constructive feedback when needed. 4.63 4.18 5.00 4.44

Overall I am satisfied with the quality of this

subject. 4.50 3.95 4.80 4.25

I feel that the rules for referencing, late

penalties and extensions were consistently

applied according to Faculty Guidelines.

4.25 4.00 4.60 4.16

I was provided with consistent advice about

assessment expectations. 4.44 4.27 5.00 4.41


November 2012

NVivo coding of student comments

Steps to follow: 1. Raw comment file 2. Mail merge for import 3. NVivo import 4. Node creation 5. Coding 6. Interpretation and write up


November 2012

Subject Code Ranking information Subj1 Subj 2 Subj 3 Subj 4 Subj 5 Subj 6

Subject Name Av % No % Not Dom

Qualitative respondents (N) N=50 N=29 N=36 N=39 N=39 N=66

Percentage of total respondents (quantitative) 10.8% 14.0% 18.8% 34.8% 26.4% 24.7%

Confidence Level LOW LOW LOW LOW LOW MODERATE

1. False positives 27% 4 0 12% 40% N/A N/A 36% 19%

2. Interesting subject matter 26% 4 0 24% N/A 43% 14% N/A 21%

3. Practical aspects 16% 3 0 N/A N/A N/A 22% 14% 12%

4. Relevance to future careers and industry 18% 2 2 N/A Not dominant N/A 22% 14% Not dominant

5. Positive re. teaching staff 17% 2 2 Not dominant N/A 22% 12% Not dominant N/A

6. Field trips 20% 1 0 20% N/A N/A N/A N/A N/A

7. Learning about new technology in the industry 18% 1 1 N/A N/A N/A 18% Not dominant N/A

8. Good assessment tasks 16% 1 1 N/A N/A 16% N/A Not dominant N/A

9. Tutorials/Lab Sessions 15% 1 0 N/A 15% N/A N/A N/A N/A

10. Cost X software 0% 0 1 N/A Not dominant N/A N/A N/A N/A

11. Well structured (2009) 0% 0 1 N/A N/A N/A N/A N/A Not dominant

12. Skill development opportunities 0% 0 1 N/A N/A N/A N/A N/A Not dominant

Different ways of reporting on the results: 2: Compare and contrast main themes across subjects or courses


November 2012

3: Visualising qualitative data using word clouds http://worditout.com/word-cloud/make-a-new-one


November 2012

Common themes which emerge from student

comments at UTS

Positive Comments – The top five common themes

1. Positive comments regarding some of the teaching staff

2. Practical elements of the subject/course (hands on tasks)

3. Relevance of content to industry and future careers (e.g., up to

date, industry standard technology and practice)

4. ‘False positives’

5. Industry placements (e.g., valuable experience, networking and

contacts)

Other themes not in the top five include: having a general interest in

the subject content and finding it enjoyable (self-selection), positive

experiences with group work, receiving timely and useful feedback,

quality of assessment tasks, and the interesting guest lecturers.


November 2012

Common themes which emerge from student

comments at UTS

Negative or ‘Room for Improvement’ Comments

The top five common themes

1. ‘Variation in the quality of the teaching staff’

2. More industry relevant, practical work

3. Better organised on course/subject administration: i.e. uploading lecture

materials, monitoring discussion boards, responding to queries, etc.

4. More targeted, individual and timely feedback on assessment tasks

5. Structure of subject/course in need of updating/review - e.g., poorly timed

assessment tasks, poor weighting of tasks

Other themes not in the top five include: improving the quality of (or training for)

the tutoring staff, more Summer and block-mode classes for part-time/working

students, better utilisation of laboratory and simulation facilities


November 2012

Important issues to consider when dealing with student feedback

Group task: Yellow ‘Scenario Cards’ Go to page 6 of your workbook

• What would you do if you came across these comments? • Rate the severity of these comments and whether action needs to be taken • What other information might you need in order to make a judgement? • What are the policies and procedures that you think might apply in this

situation? • What would the ideal outcome of the scenario be?


November 2012

Overview of how student comments are used to improve learning and teaching at UTS

INPUTS Inst. Interactive Media and Learning Faculty

Liaison Officers

Faculty Exec Staff Course Review Advisory Group ‘Watch-list’

PQU (Me)

ACCESS Vice Chancellor, D. Vice

Chancellor Teaching and Learning

Inst. Interactive Media and Learning Faculty

Liaison Officers

Associate Deans Teaching and Learning

Course and Subject Coordinators

USES $$ Promotions and pay

decisions $$ Awards for quality

teaching and learning

!!!!! CHANGE!!!!

new approaches to course and subject improvement through ...aair.org.au/app/webroot/media/pdf/aair...

Documents