learning outcomes assessment of master and phd...
TRANSCRIPT
![Page 1: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/1.jpg)
Learning Outcomes Assessment of Master and PhD Theses
Kamran SiddiquiMechanical and Materials Engineering, Western
University
Learning Outcomes: Evolution of Assessment DoubleTree by Hilton Hotel, Toronto
October 17-18, 2016
Mechanical and Materials Engineering
![Page 2: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/2.jpg)
Introduction Master and PhD theses comprise a major
portion of graduate curriculum A strong indicator of,
• technical skills, • thinking, analytical and writing/presentation
abilities A key parameter in the overall assessment of
graduate degree outcomes Large variation in the standards used by
faculty members to judge theses
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 3: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/3.jpg)
Introduction Thesis work covers all six graduate attributes
identified by the Council of Ontario Universities
Hence, a strong indicator of the fulfilment of degree level expectations
Currently, scarcity in the use of quality indicators to assess graduate attributes of theses
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 4: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/4.jpg)
Workshop focus Learning outcomes assessment of graduate
theses Topics of discussion,
• Development of assessment rubric• Its implementation• Data collection and analysis • Thesis self-assessment• Future directions
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 5: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/5.jpg)
Ontario Universities’ Degree Level Expectations
Depth and breadth of knowledge
Research and scholarship
Application of knowledge
Autonomy and professional capacity
Communication skills
Awareness of the limits of knowledge
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 6: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/6.jpg)
Initiative and challenges Graduate program in Mechanical and
Materials Engineering initiated the process of learning outcomes assessment in Winter 2015
Used available resources to develop a rubric for graduate thesis assessment
A major challenge was to keep the rubric general enough to accommodate research diversity
Should cover the overall research expectations i.e. thesis content and oral defense
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 7: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/7.jpg)
Further challenges Thesis quality indicators and related rubrics in
the literature are primarily defined based on thesis skeleton i.e., • Introduction• Literature review• Theory• Methods• Results/data analysis• Discussion and conclusions
No indicators and rubrics for thesis assessment are defined based on six graduate attributes
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 8: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/8.jpg)
Approach used Consulted Graduate Degree Level Expectations
guidelines (OUCQA) Changed from 4-point to 3-point indicators Research and scholarship divided into two sub-
categories• General• Critical thinking
Communication skills divided into two sub-categories• Thesis• Oral exam
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 9: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/9.jpg)
Developed Rubric
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 10: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/10.jpg)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 11: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/11.jpg)
Implementation
Started from April 2015
Thesis + oral defense assessed by all examiners
Each examiner assigned a numeric score against each attribute
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 12: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/12.jpg)
DBK: Depth and breadth of knowledge RS(G): Research and scholarship (General)RS(CT): Research and scholarship (Critical thinking) AK: Application of knowledgePC: Professional capacity CS(T): Communication skills (Thesis)CS(O): Communication skills (Oral exam) ALK: Awareness of limits of knowledge
Data Analysis(Sample size=33, PhD=3, Master=30)
1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
DBK RS(G) RS(CT) AK PC CS(T) CS(O) ALK Overall
Average Scores (Mechanical Engineering Graduate Students)
Master
PhD
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 13: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/13.jpg)
Individual scores in each graduate attribute
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
DBK
R&S(G)
R&S (CT)
AK
PC
CS (T)
CS (E)
ALK
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 14: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/14.jpg)
2.0
2.1
2.2
2.3
2.4
2.5
2.6
DBK RS(G) RS(CT) AK PC CS(T) CS(O) ALK Overall
Canadian International
Categorization based on prior degree
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 15: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/15.jpg)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Canadian
DBK
RS(G)
RS(CT)
AK
PC
CS(T)
CS(O)
ALK
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1 2 3 4 5 6 7 8 9 10 11 12 13
International
DBK
RS(G)
RS(CT)
AK
PC
CS(T)
CS(O)
ALK
Individual scores based on prior degree
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 16: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/16.jpg)
Thesis self-assessment The developed rubric serves as an
assessment tool for examiners Students are aware of this rubric No formal mechanism in place to self assess
thesis expectations against the rubric A thesis self-assessment form is recently
introduced Students self-assess their theses against the
rubric indicators
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 17: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/17.jpg)
Thesis self-assessment Students complete this form and submit
along with thesis for examination Key benefits,
• Students can judge the strengths and weaknesses in their theses
• Assist in the preparation of thesis defense• Provides the examiner a snoop of thesis quality in
meeting the expectations before a thorough thesis review
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 18: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/18.jpg)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 19: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/19.jpg)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 20: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/20.jpg)
Example
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 21: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/21.jpg)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 22: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/22.jpg)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 23: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/23.jpg)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 24: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/24.jpg)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 25: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/25.jpg)
Future directions
Separate rubrics for Master and Doctoral theses
Collection and analysis of the assessment data
Identification of any shortcoming and necessary corrective action
Incorporation into the overall curriculum mapping of the graduate programs
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 26: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/26.jpg)
References Lovitts, B. E. (2007). Making the Implicit Explicit: Creating
Performance Expectations for the Dissertation. Stylus Publishing, LLC.
Lovitts, B. E., & Wert, E. L. (2008). Developing Quality Dissertations in the Sciences: A Graduate Student’s Guide to Achieving Excellence. Stylus Publishing, LLC.
Maki, P., & Borkowski, N. A. (2006). The Assessment of Doctoral Education: Emerging Criteria and New Models for Improving Outcomes. Stylus.
Appendix 1: OCAV’s Undergraduate and Graduate Degree Level Expectations, Ontario Universities Council on Quality Assurance (http://oucqa.ca/framework/appendix-1/)
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 27: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/27.jpg)
Questions for table discussion Strengths and shortcomings of the thesis
assessment rubric Other aspects of data analysis and
comparison Utilization of results from the collected
data Strengths and shortcomings of students’
thesis self-assessment
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 28: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/28.jpg)
Thank you
Learning Outcomes: Evolution of Assessment Toronto, October 17-18, 2016
![Page 29: Learning Outcomes Assessment of Master and PhD …oucqa.ca/.../03/Learning-Outcomes-Assessment-of-Master-and-Ph… · Learning Outcomes Assessment of Master and PhD Theses. ... Thesis](https://reader035.vdocuments.net/reader035/viewer/2022081401/5ade3bea7f8b9a8b6d8dfb26/html5/thumbnails/29.jpg)