2014-15 budget measure
DESCRIPTION
Quality Indicators for Learning and Teaching 2014 Survey Managers’ Information Forum 18 July 2014 Dr Andrew Taylor Branch Manager, Higher Education Data and Analysis. 2014-15 Budget Measure. 2014-15 Budget measure Upholding Quality – Quality Indicators for Learning and Teaching (QILT) - PowerPoint PPT PresentationTRANSCRIPT
Quality Indicators for Learning and Teaching
2014 Survey Managers’ Information Forum18 July 2014
Dr Andrew TaylorBranch Manager, Higher Education Data and Analysis
2014-15 Budget Measure
• 2014-15 Budget measure Upholding Quality – Quality Indicators for Learning and Teaching (QILT)
• QILT – suite of surveys over the student life cycle, from commencement to employment
• New, purpose built website to publish results and inform student choice
• Indicator development for performance funding
• AQHE Reference Group, chaired by Professor Ian O’Connor
• Consultation process:
– Discussion papers
– Submissions
– Roundtables
Development of Indicators
• Survey burden on students and institutions
• Centralised administration
• Employer surveys ad hoc and not well done
• Need for robust measures of teaching performance
Key issues
AQHE Report
Key Reference Group recommendations:• Centralised administration of the suite of performance
measurement instruments under a single third-party provider• The suite of instruments to initially comprise the University
Experience Survey (UES) and a new Graduate Outcomes Survey (GOS)
• Scoping study on a new instrument to assess employer needs and satisfaction with graduates
Not recommended:• Composite Teaching Quality Indicator• Collegiate Learning Assessment
MyUniversity website
• Live in April 2012• Clear, meaningful and transparent information• Confidence intervals• Subject by institution level data
• Amendments made to the Higher Education Support Act 2003 (HESA) in November 2013
• Personal information can now be provided to a range of bodies, including– TEQSA– higher education providers and some peak bodies– state governments
• Safeguards remain– Limited uses– No on-sharing– Provider consent
• Universities Australia data sharing agreement allows access for member institutions
Data sharing
QILT Components• Three surveys:
University Experience Survey (UES)Graduate Outcomes Survey (GOS)Employer Satisfaction Survey (ESS)
• Plus new purpose built QILT website to publish results from the surveys
• QILT contractor selected through tender to administer all elements
Timing
2014 2015 2016 2017
University Experience Survey
August August August August
Graduate Outcomes Survey
n/a October round April and October rounds
April and October rounds
Employer Satisfaction Survey
October June June June
Website August – initial release
February and August updates
February and August updates
February and August updates
University Experience Survey
• 2014 UES administered by GCA/SRC consortium • 2015 UES and 2016 UES comprise part of QILT• Methodology in line with previous administrations
National results, key items, 2011-2013
2011 2012 201350
55
60
65
70
75
80
85
90
Percentage satisfied, quality of teaching item
Commencing Later Year Total
2011 2012 201350
55
60
65
70
75
80
85
90
Percentage satisfied, quality of en-tire education experience item
Commencing Later Year Total
National results, scales, 2012-2013
Teaching Quality Learner Engagement Learning Resources Student Support Skills Development0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
81
58
82
53
8279
57
83
53
79
Percentage satisfied, UES focus areas
2012 2013
International benchmarking
2008 2009 2010 2011 2012 201370
72
74
76
78
80
82
84
86
88
90
Entire educational experience rated positively, UES and NSSE, 2008-2013
NSSE 1st year NSSE senior year UES 1st year UES later year
Perc
enta
ge s
atisfi
ed
Graduate Outcomes Survey
• GOS will provide timely labour market information• Review existing AGS instrument to develop new GOS
instrument and methodology • Reporting methods need review• Survey administered in two rounds October and April
(following year), though alternative proposals considered• 2015 AGS administered by GCA• GOS to replace AGS from the second half of 2015 • 2016 GOS and 2017 GOS comprise part of QILT
Employer Satisfaction Survey• ESS measures employer satisfaction with generic skills, technical
skills and work readiness of graduates• Recent graduates provide contact details of direct supervisor• Pilot ESS in 2013-14 across 4 universities and 5 fields of education• Pilot ESS report available from: www.education.gov.au/employer-satisfaction-survey• Further development of ESS in October-December 2014
instrument and survey methodology to be reviewed and tested on larger scale
universities participate opt-in basis• 2014 ESS trial, 2015 ESS and 2016 ESS comprise part of QILT
ESS – key resultsOverall 92 per cent of employers stated they would be confident in recommending someone with the same qualification from the same university for a similar position to the graduate’s role
Overall satisfaction by key factors (skill clusters) Foundation skills — 95 per cent Adaptive skills — 89 per cent Teamwork and interpersonal skills — 96 per cent Disciplinary skills — 93 per cent Employability skills — 93 per cent Enterprise skills — 79 per cent
QILT Website• QILT website aims to inform student choice through provision of survey
results by institution and course • Two phases:1. Initial release
August 2014 release Existing data from the AGS and UES, to be provided by department Limited scope – undergraduates at Table A and B universities
2. Further development Developed progressively to August 2015 Expanded scope – all institutions and levels of study Enhanced functionality, e.g. search and comparison functions
enhanced Development and release of mobile app
• Website updated every six months from August 2014 with latest available QILT survey data
Governance
• QILT Working Group will provide advice to department on implementation of QILT
• QILT Working Group largely comprises former members of AQHE Reference Group
• QILT Working Group will include representation from non-university higher education institutions (NUHEIs)
General methodology
• Independent administration• Random stratified sample• Stratification variables are specified for each instrument• HEIMS data to be made available for constructing sample
frames• Additional information for the construction of sample frames,
along with student and graduate contact details, to be collected from institutions
Quality assurance• Biases and unrepresentativeness minimised• Quality assurance is based on achieving required confidence
interval, on specific items, for a set proportion of strata• UES (entire educational experience) and GOS (further full-
time study)– Interval of + or – 7.5 pp, at 90% confidence level– 75% of strata at the institution by subject area level,
excluding strata with a population of fewer than six• ESS (overall supervisor rating)– Interval of + or – 7.5 pp, at 90% confidence level– 8 out of 10 broad fields of education– 25 responses for each field at each institution
• Confidence intervals calculated as per 2013 UES
NUHEIs
• Currently around 85 Non University Higher Education Institutions (NUHEIs) accredited under HESA
• Limited participation in AGS to date, not in scope for UES• NUHEIs in scope for QILT:
2015 ESS 2015 UES (trial of NUHEIs in 2014 UES)2016 GOS (from October 2015 round)
• Survey results for NUHEIs to be included on QILT website subject to data being robust
Tender• Tender opened 13 June• Tender closed 11 July• Professor Shirley Alexander, UTS, to assist
Department in assessing tenders• Expect to announce outcome by the end of July• Market testing through tender for 2014-16 triennium
23
CONFIDENCE INTERVAL PAPER - ANU
http://unistats.anu.edu.au/surveys/toolkit/Confidence_Intervals_Sheet.pdf