First-Assessment First-First-Assessment First-Feedback : A “just in time” Feedback : A “just in time”

Intervention to Support At-risk Intervention to Support At-risk First Year StudentsFirst Year Students

Profs Alf Lizzio & Keithia WilsonProfs Alf Lizzio & Keithia WilsonGriffith UniversityGriffith University

Acknowledgment of CountryAcknowledgment of Country

In the Spirit of ReconciliationIn the Spirit of ReconciliationFollowing on from Sorry DayFollowing on from Sorry DayI would like to acknowledge & honour I would like to acknowledge & honour the Traditional Custodians of the land the Traditional Custodians of the land that we are meeting on today, the that we are meeting on today, the Yugambeh People, and pay respect Yugambeh People, and pay respect to their Elders past & presentto their Elders past & present

RATIONALE: Why is this RATIONALE: Why is this important?important?

Students’ early experiences of university directly influence both Students’ early experiences of university directly influence both their ongoing their ongoing learning outcomes & persistencelearning outcomes & persistence..

Students’ Students’ performance on assessment,performance on assessment, especially their first especially their first assessment itemassessment item is most influential in this regard. is most influential in this regard.

How well students perform on their early assessment tasks can How well students perform on their early assessment tasks can initiate either a initiate either a virtuousvirtuous (building confidence) or (building confidence) or viciousvicious (decreasing confidence) academic cycle. (decreasing confidence) academic cycle.

Students who fail, ‘just pass’ or who ‘do worse than expected’ on Students who fail, ‘just pass’ or who ‘do worse than expected’ on early assessment are likely to suffer a early assessment are likely to suffer a loss of personal & loss of personal & academic confidenceacademic confidence (Zajacova & Espenshade, 2005). (Zajacova & Espenshade, 2005).

This performance feedback may also signal This performance feedback may also signal a gap in study skills a gap in study skills and practices, self-management capability or academic and practices, self-management capability or academic efficacyefficacy that may be amenable to early intervention & that may be amenable to early intervention & improvement. improvement.

RATIONALE: Can students RATIONALE: Can students manage what is expected? manage what is expected?

First year students often do not possess First year students often do not possess sufficient self-sufficient self-regulation and problem-solving capacitiesregulation and problem-solving capacities to adequately to adequately prepare for, or process these potentially challenging prepare for, or process these potentially challenging experiences, with implications for their subsequent academic experiences, with implications for their subsequent academic engagement, learning outcomes and persistence. engagement, learning outcomes and persistence.

For example, recent research points to a For example, recent research points to a lack of fit orlack of fit or incongruenceincongruence between staff and commencing students’ between staff and commencing students’ (mis)-conceptions & expectations of assessment tasks (e.g., (mis)-conceptions & expectations of assessment tasks (e.g., What’s involved? How best to prepare? What investment is What’s involved? How best to prepare? What investment is required? What help is available?) (Collier & Morgan, 2008).required? What help is available?) (Collier & Morgan, 2008).

This is even more likely to be the case with This is even more likely to be the case with first-generation first-generation university studentsuniversity students who, by virtue of their circumstances, who, by virtue of their circumstances, may possess significantly less cultural capital and academic may possess significantly less cultural capital and academic resourcefulness (Hattie & Timperley, 2007).resourcefulness (Hattie & Timperley, 2007).

INTERVENTION AIMS: What INTERVENTION AIMS: What difference are we trying to make?difference are we trying to make?

The FAFF process is designed as an The FAFF process is designed as an academic recoveryacademic recovery, just-in-, just-in-time intervention to develop self-management and problem-solving time intervention to develop self-management and problem-solving capabilities in first year students.capabilities in first year students.

Help is initially offered in response to an identified ‘academic crisis’ Help is initially offered in response to an identified ‘academic crisis’ or at a period where students are more likely to be responsive or at a period where students are more likely to be responsive because they are engaged with a real problem, namely an because they are engaged with a real problem, namely an ‘unexpected result’ on their first assessment item.‘unexpected result’ on their first assessment item.

First-Assessment First-Feedback particularly First-Assessment First-Feedback particularly targetstargets at-risk first at-risk first year students who failed or marginally passed their first piece of year students who failed or marginally passed their first piece of university assessment.university assessment.

INTERVENTION PROCESS: How INTERVENTION PROCESS: How do we do it?do we do it?

An example of “intrusive academic advising” (Earl, 2006)An example of “intrusive academic advising” (Earl, 2006)

The intervention involves studentsThe intervention involves students::1 Completing a 1 Completing a reflective workbookreflective workbook to help them to help them

understand the reasons for their performance and to understand the reasons for their performance and to identify improvement goals and strategies. identify improvement goals and strategies.

2 Participating in an intensive 2 Participating in an intensive guided discussionguided discussion with their with their tutor tutor

3 Developing an 3 Developing an academic recovery action planacademic recovery action plan4. Negotiating 4. Negotiating follow-upfollow-up to maintain momentum. to maintain momentum.

EVIDENCE BASE: Where is EVIDENCE BASE: Where is the point of optimal leverage?the point of optimal leverage?

Recent cross-institutional multi-level analysis on Recent cross-institutional multi-level analysis on the efficacy of transitional or preparatory the efficacy of transitional or preparatory programs for first-year students (Porter & Swing, programs for first-year students (Porter & Swing, 2006) indicates that 2006) indicates that a more focused approacha more focused approach linked to specific academic tasks in a particular linked to specific academic tasks in a particular disciplinary context may be the most effective disciplinary context may be the most effective way to impact on academic capability and way to impact on academic capability and persistence. persistence.

THEORETICAL FRAMEWORK:THEORETICAL FRAMEWORK:WHAT IS SELF-REGULATION?WHAT IS SELF-REGULATION?

Self-regulationSelf-regulation - a set of metacognitive, behavioural - a set of metacognitive, behavioural and motivational strategies that learners can use to and motivational strategies that learners can use to control their learning processes (Zimmerman, 1990) & control their learning processes (Zimmerman, 1990) & actively manage their own learning outcomes (Pintrich actively manage their own learning outcomes (Pintrich (1999). (1999). Self-regulation is particularly Self-regulation is particularly required at times of required at times of change, stress or transitionchange, stress or transition where a person is required where a person is required to respond to new demands & where automatic or to respond to new demands & where automatic or routine responses are not sufficient. routine responses are not sufficient. Thus self-regulation is particularly salient in higher Thus self-regulation is particularly salient in higher education contexts because of the (often implicit) education contexts because of the (often implicit) expectation of independenceexpectation of independence placed upon placed upon commencing students. commencing students.

What influences students’ What influences students’ capacity to self-regulate?capacity to self-regulate?

Self-management of academic Self-management of academic performance can be understood as a performance can be understood as a function of a function of a complex arraycomplex array of of interdependent personal & environmental interdependent personal & environmental factorsfactors

ENVIRONMENTAL FACTORS: ENVIRONMENTAL FACTORS: What is the ‘contextual swim’ What is the ‘contextual swim’

students are in?students are in?

Temporal context: Interaction of past, present & Temporal context: Interaction of past, present & anticipated future factors anticipated future factors

Sense of Sense of pastpast learning experiences learning experiences type & type & level of social capital students bringlevel of social capital students bring

Sense of Sense of futurefuture (personal & career) & (personal & career) & motivation for being at universitymotivation for being at university

Sense of Sense of presentpresent environment – balancing environment – balancing university & work-life university & work-life

PERSONAL FACTORSPERSONAL FACTORSWhat is the ‘personal swim’ What is the ‘personal swim’

students are in?students are in?

Four factors have been found to affect self-Four factors have been found to affect self-regulation & influence level of task engagement regulation & influence level of task engagement & persistence:& persistence:Beliefs about learning & knowledge influence Beliefs about learning & knowledge influence learning strategieslearning strategiesAcademic efficacy or expectations about Academic efficacy or expectations about successful task accomplishmentsuccessful task accomplishmentLevel of achievement motivation Level of achievement motivation Help-seeking behaviourHelp-seeking behaviour

The Self-Regulation CycleThe Self-Regulation Cycle

1 Contextual Awareness

2 Goal Setting

3 StrategySelection

4 Action

5 Learning and

adaptation

Personal Factors

PastExperiences

FutureMotivations

Present Situation

Self-Regulating assessment performance: Self-Regulating assessment performance: The basic action cycleThe basic action cycle

1.1. Contextual awareness - understanding demands & Contextual awareness - understanding demands & constraints of the assessment taskconstraints of the assessment task

2.2. Goal setting – prioritising what has to be done at what Goal setting – prioritising what has to be done at what personal standardpersonal standard

3.3. Strategy selection – organising time & choosing Strategy selection – organising time & choosing appropriate study strategies for the taskappropriate study strategies for the task

4.4. Action – managing distractions, avoidances etc. in a Action – managing distractions, avoidances etc. in a timely way to produce a quality producttimely way to produce a quality product

5.5. Learning & adaption – reflecting on outcomes & Learning & adaption – reflecting on outcomes & feedbackfeedback

SELF-REGULATION ACTION SELF-REGULATION ACTION CYCLE & ASSESSMENTCYCLE & ASSESSMENT

Therefore, failure on assessment can be Therefore, failure on assessment can be understood as a understood as a failure of academic self-failure of academic self-regulationregulation or a breakdown at one or more or a breakdown at one or more stages of the above action cyclestages of the above action cycle

ACADEMIC RECOVERY & ACADEMIC RECOVERY & SELF-REGULATIONSELF-REGULATION

From this perspective ‘academic recovery’ requires the From this perspective ‘academic recovery’ requires the student to become somewhat meta-cognitive about ‘what student to become somewhat meta-cognitive about ‘what happened’ and ‘what should be done differently’ to happened’ and ‘what should be done differently’ to ensure better future outcomes. ensure better future outcomes.

While many students are able to evidence this process While many students are able to evidence this process of reflection on and learning from experience (viz., of reflection on and learning from experience (viz., thinking about their thinking), those most at-risk may be thinking about their thinking), those most at-risk may be least able to do this. least able to do this.

The emotional consequences of failure & the competing The emotional consequences of failure & the competing and complex demands of a new environment may and complex demands of a new environment may combine to inhibit problem solving and help-seeking combine to inhibit problem solving and help-seeking behaviour behaviour

WORKBOOK DESIGNWORKBOOK DESIGN

The Workbook is designed to The Workbook is designed to Follow the logic of self-regulation (viz., the Follow the logic of self-regulation (viz., the ‘medium is the message’)‘medium is the message’)Prime the advising interviewPrime the advising interview

The process is generically applicable to:The process is generically applicable to:Any disciplinary contextAny disciplinary contextAny student cohortAny student cohortA wide range of types of assessmentA wide range of types of assessment

WORKBOOK STRUCTUREWORKBOOK STRUCTURE

1.1. My foundationMy foundation: How do I get off to a good start? Student’s : How do I get off to a good start? Student’s readiness to engage in problem solvingreadiness to engage in problem solving

2.2. My assessmentMy assessment: What is my current situation?: What is my current situation?

3.3. My understandingMy understanding: What do I think is going on?: What do I think is going on?

4.4. My study profileMy study profile: What do I think is going on?: What do I think is going on?

5.5. Reflecting on written feedbackReflecting on written feedback: What did staff tell me about my : What did staff tell me about my work?work?

6.6. Future performancesFuture performances: What am I expecting?: What am I expecting?

7.7. Goal settingGoal setting: What is the best way forward?: What is the best way forward?

8.8. Action planning:Action planning: How do I put my goals into action? How do I put my goals into action?

9.9. Closure & reviewClosure & review: How useful was this exercise?: How useful was this exercise?

Facilitating self-understanding of Facilitating self-understanding of assessment performance: The domains of assessment performance: The domains of

potential influencepotential influence

Step 3 – My understanding: What do I think is going on?Step 3 – My understanding: What do I think is going on?Academic confidence:Academic confidence: Do I think I can do this? Do I think I can do this?Academic motivation:Academic motivation: Do I want to do this? Do I want to do this?Beliefs about Learning:Beliefs about Learning: How do I approach learning? How do I approach learning?Awareness:Awareness: Do I know what is expected? Do I know what is expected?Commitment:Commitment: Do I set myself study goals? Do I set myself study goals?Study Strategies:Study Strategies: Do I study effectively? Do I study effectively?Actions:Actions: Do I put my plans into action? Do I put my plans into action?Help-seeking:Help-seeking: Can I get the help I need to succeed? Can I get the help I need to succeed?Present Circumstances:Present Circumstances: What is happening in my life? What is happening in my life?Sense of University:Sense of University: What’s it like for me here? What’s it like for me here?History:History: Where have I come from? Where have I come from?Future:Future: Where am I headed? Where am I headed?

INTERVENTION TARGETINTERVENTION TARGET

FY students determined to be at-riskFY students determined to be at-risk

All first year students who fail or ‘just pass’ All first year students who fail or ‘just pass’ their first piece of assessment in the their first piece of assessment in the threshold course for semesters 1 and 2threshold course for semesters 1 and 2

Band of 0 – 60 markBand of 0 – 60 mark

INTERVENTION INTERVENTION COMPONENTSCOMPONENTS

1.1. Students are contacted by tutors via, email & Students are contacted by tutors via, email & phone, and invited to participate in the phone, and invited to participate in the process.process.

2.2. Students complete a reflective workbook Students complete a reflective workbook structured around a problem-solving cycle structured around a problem-solving cycle (viz., facilitating readiness, self-assessment (viz., facilitating readiness, self-assessment of performance gaps, clarification of efficacy of performance gaps, clarification of efficacy expectations, goal setting and action planning) expectations, goal setting and action planning) designed to help them understand the reasons designed to help them understand the reasons for their performance on early assessment and for their performance on early assessment and identify improvement goals and strategies.identify improvement goals and strategies.

INTERVENTION INTERVENTION COMPONENTSCOMPONENTS

3. Students then meet with a tutor and 3. Students then meet with a tutor and participate in a guided discussion based participate in a guided discussion based on the workbook which concludes with on the workbook which concludes with action planning and where appropriate, action planning and where appropriate, linkage/referral to university resources.linkage/referral to university resources.

4. Tutor and student ‘stay in contact’ (e.g., 4. Tutor and student ‘stay in contact’ (e.g., pre-arranged phone or email contact) to pre-arranged phone or email contact) to maintain positive momentum.maintain positive momentum.

EVALUATIONEVALUATION

Students complete an evaluation survey containing both Students complete an evaluation survey containing both rating scales (1-7) and open-ended questions focused rating scales (1-7) and open-ended questions focused on their on their experiences experiences of the process and of the process and outcomes outcomes of of the intervention.the intervention.The subsequent The subsequent academic performanceacademic performance of students of students who undertook the intervention (n = 30) was compared who undertook the intervention (n = 30) was compared to students of similar achievement in the same courses to students of similar achievement in the same courses who did not participate in the intervention (n = 45).who did not participate in the intervention (n = 45).Students’ responses to the workbook were analysed for Students’ responses to the workbook were analysed for key key themesthemes related to their understanding of their related to their understanding of their underperformance.underperformance.

OUTCOMES: Academic OUTCOMES: Academic Persistence Persistence

Improved student persistence: Improved student persistence:

90% of students who participated in the 90% of students who participated in the intervention submitted their second piece intervention submitted their second piece of assessment compared to a base of assessment compared to a base submission rate of 78% of comparable submission rate of 78% of comparable students who did not participate in the students who did not participate in the interventionintervention

OUTCOMES: Academic OUTCOMES: Academic successsuccess

Improved academic successImproved academic success

100% of students who participated passed 100% of students who participated passed their next piece of assessment compared their next piece of assessment compared to a base pass rate of 77% of students of to a base pass rate of 77% of students of comparable academic standard who did comparable academic standard who did not participate in the intervention not participate in the intervention

OUTCOMES: Passing the OUTCOMES: Passing the coursecourse

Improved overall academic success Improved overall academic success

60% of students who participated in the 60% of students who participated in the intervention passed the course compared intervention passed the course compared to only 24% in the non-intervention to only 24% in the non-intervention comparison group.comparison group.

OUTCOMES: Student OUTCOMES: Student EvaluationsEvaluations

At-risk students’ self-reported evaluations of the At-risk students’ self-reported evaluations of the process and outcomes of the intervention were process and outcomes of the intervention were uniformly positiveuniformly positive. .

Students rated the intervention as producing high Students rated the intervention as producing high levels of :levels of :academic related learning (mean = 5.7/7, sd academic related learning (mean = 5.7/7, sd = .68) and = .68) and personal development (mean = 5.02, sd = .62). personal development (mean = 5.02, sd = .62).

OUTCOMES: Student Process OUTCOMES: Student Process EvaluationsEvaluations

Specific improvements were reported in terms of:Specific improvements were reported in terms of:Greater Greater insightinsight into the reasons for under- into the reasons for under-performance on assessment (mean = 5.56, sd. = performance on assessment (mean = 5.56, sd. = .59).59)Increased Increased efficacy efficacy and and optimismoptimism for future for future performance (mean = 5.57, sd. .68) performance (mean = 5.57, sd. .68) Given their superior comparative performance Given their superior comparative performance reported above, it would seem that students’ reported above, it would seem that students’ enhanced sense of efficacy was well-founded. enhanced sense of efficacy was well-founded.

OUTCOMES: Student Process OUTCOMES: Student Process EvaluationsEvaluations

Importantly, given their at-risk status Importantly, given their at-risk status students also reported the process itself students also reported the process itself as as non-aversive non-aversive (mean = 5.31, sd = .74).(mean = 5.31, sd = .74).

OUTCOMES: Tutor EvaluationsOUTCOMES: Tutor Evaluations

As a result of the intervention tutors As a result of the intervention tutors reported: reported:

A stronger relationship with studentsA stronger relationship with students

Higher attendance by those students at Higher attendance by those students at tutorialstutorials

Greater student engagementGreater student engagement

STUDENT EVALUATIONS - Mechanisms STUDENT EVALUATIONS - Mechanisms contributing to enhanced performance and efficacycontributing to enhanced performance and efficacy

The FAFF intervention clearly functions at socio-The FAFF intervention clearly functions at socio-emotional, task specific and general self-emotional, task specific and general self-regulatory levels. Students described -regulatory levels. Students described -The positive value of feeling normalised (The positive value of feeling normalised (I I thought it was only methought it was only me); ); Experiencing positive regard and support (Experiencing positive regard and support (The The experience of somebody caring helped me to experience of somebody caring helped me to feel better about myselffeel better about myself););Cueing help-seeking (Cueing help-seeking (I wouldn’t have done I wouldn’t have done anything if you hadn’t reached outanything if you hadn’t reached out); and ); and The value of problem solving and goal setting ( The value of problem solving and goal setting ( I I needed this structureneeded this structure). ).

STUDENT EVALUATIONS: Ongoing STUDENT EVALUATIONS: Ongoing ChallengesChallenges

The high ratings for :The high ratings for :fixed-ability related concerns (fixed-ability related concerns (I’m not smart I’m not smart enoughenough) in students’ explanations of their under-) in students’ explanations of their under-performance (fail or near fail) on the first piece of performance (fail or near fail) on the first piece of assessmentassessmentcompared to effort and organisation related compared to effort and organisation related concerns (concerns (I didn’t work hard enough) I didn’t work hard enough) also indicate the fundamental challenge in also indicate the fundamental challenge in facilitating movement from an facilitating movement from an ego to a learning ego to a learning orientationorientation (Kluger & DeNisi, 1996). (Kluger & DeNisi, 1996).

STUDENT EVALUATIONS: Meta-STUDENT EVALUATIONS: Meta-themesthemes

Consistent with this, two Consistent with this, two meta-themesmeta-themes were were apparent from student feedback about the apparent from student feedback about the efficacy of the intervention: efficacy of the intervention:

1.1. an improved capacity to clearly an improved capacity to clearly appraise appraise their their academic performance; and academic performance; and

2.2. a shift from an anxiety-based, ego orientation a shift from an anxiety-based, ego orientation to a to a problem-solving task orientationproblem-solving task orientation. .

NEXT STEPSNEXT STEPS

2008-09 - Currently implementing the 2008-09 - Currently implementing the FAFF intervention across Griffith Health FAFF intervention across Griffith Health Group (Griffith L&T Grant)Group (Griffith L&T Grant)2009-10 - Will be testing the efficacy and 2009-10 - Will be testing the efficacy and generalisability of the FAFF Workbook generalisability of the FAFF Workbook intervention across other disciplines intervention across other disciplines (business & law) in another university (business & law) in another university (Newcastle) funded by an ALTC Priority (Newcastle) funded by an ALTC Priority Grant.Grant.

2009-2010: From Early Intervention 2009-2010: From Early Intervention to Preventionto Prevention

Developing & testing a Developing & testing a First-Assessment First-Success First-Assessment First-Success WorkbookWorkbook to provide all commencing students with a structured to provide all commencing students with a structured process to help them orient, engage and prepare for their process to help them orient, engage and prepare for their early/first assessment tasks. early/first assessment tasks.

The workbook process will firstly raise students’ metacognitive The workbook process will firstly raise students’ metacognitive awareness of the salient personal (e.g., study strategies and awareness of the salient personal (e.g., study strategies and attitudes, motivations, help-seeking behaviour, etc.) and attitudes, motivations, help-seeking behaviour, etc.) and environmental (e.g., staff expectations, task demands, etc.) environmental (e.g., staff expectations, task demands, etc.) factors likely to impact on their engagement and success with factors likely to impact on their engagement and success with their first assessment task. their first assessment task.

Students will then work through the self-regulation cycle (viz., Students will then work through the self-regulation cycle (viz., awareness, goals, strategies, action) to develop a considered awareness, goals, strategies, action) to develop a considered first assessment management plan.first assessment management plan.