Making it count Reflecting on the National Student Survey in the process of enhancement Alex Buckley

2

3

Contents

Section

Foreword

1 Introduction 1.1Aboutthisreport 1.2AbouttheInstitutionalWorkingGroup 1.3Abouttheauthor 1.4Acknowledgements

2 Staff-student partnerships 2.1Studentrepresentation 2.2Studentexploration 2.3Closingtheloop

3 Institutional structuress 3.1Staffattitudes 3.2Integrationintodecision-making 3.3Impetusforchange

4 Analysis and exploration 4.1Limitationsandchallenges 4.2Triangulationwithotherdata 4.3Benchmarking 4.4Usingqualitativedata

5 Summing up and going further 5.1Suggestionsforgoodpractice 5.2Questionsforreflection 5.3Furtherreading 5.4AdditionalHEAresources

Appendix A: Information about the NSS

Appendix B: Core NSS items

Page

4

810111212

13131923

28283335

3939444852

5656596163

65

66

4

Foreword

The question is what you do with it…

Sinceitsintroductionin2005,theNationalStudentSurvey(NSS)hasbeensubjecttoextensivedebateanddiscussion.Whiletherehavebeenconcernsaboutthevalidityofquestions,therelevanceofstudentsatisfactionasaqualityindicatorortheroleoftheNSSinleaguetablesandtheapparentlyconsequentpromotionofconsumerism,therehasequallybeenanewlevelofsectorengagementwithstudentviewsandtheroleofthesewithinlearningandteachingdevelopments.Notleastinfluencedbytheintroductionofhigherstudentfees,morechoiceforstudentsandincreasedinformationprovisionforprospectivelearners,theroleofleaguetablesmeansthattheinfluenceofNSSresultshasbecomeanaspectnoinstitutioncanaffordtoneglect.

TheNSScansupportarangeofapproachestoenhancement.Theannualproductionofleaguetablesbynewspapersisanobviousdriverforinstitutionstotakeenhancementofthestudentlearningexperienceseriously.Regardlessofwhatourethicaljudgementofthatrankingmotivationis,itiswithoutadoubthelpfulforenhancementintheclassroomifimprovementactivityissupportedandrecognisedbytheinstitution.

Morepositivelyperhaps,thecomparisonofstudentsatisfactionoveranumberofyearswithinadisciplinehashelpedusevaluatetheeffectivenessofenhancementpreviouslyintroduced.Certainlystudentcommentstoopenquestionsareofgreatusetointerpretingthestatisticaldata,whiletheadditionalbankofquestionscangiveadeeperinsightintospecificaspects,fromworkload,careersandworkplacementstocoursedelivery,or–mypersonalfavourite–feedbackfromstudents.

Andyet,inmyinstitution,themajorenhancementfromtheannualNSSresultscameperhapsnotfromthestatistics,butfromthedebateanddiscussionsthatcamefromtheevaluationofresults.Inourcase,NSS-basedenhancementactivityhasledtoasubstantialgrowthofreciprocalengagementbetweenstudentsandstaff.AnexplicitethoshasbeenestablishedthatinfluencesthequalityoflearningfarbeyondthereachoftheNSS.AtBathwetaketheviewthattherealqualityoflearningandteachinglieswhereitisowned:atthelevelofthediscipline,wherestaffandstudentsco-owntheireducation.Underpinnedbythatprinciple,wehaveengagedstaffandstudentsinalearningandteachingenhancementprocesswithoutstandingeffectsforquality.Ithasalsoresultedinoutstandinglevelsofstudentsatisfaction,puttingtheUniversityofBathconsistentlyinthetopfiveformostsatisfiedstudents.

Student engagement – the Bath approach

Studentsatisfaction–asmeasuredbytheNSS–isnotamatterthatthecorporatelevelofauniversitycaninfluencedirectly.Instead,studentsatisfactionrelatesdirectlytotheexperienceofstudentsintheirclassroomoronline,tointeractionwithteachingstaffanddepartmentalsupportstaff,tothegeneralorganisationofstudents’studiesandtointeractionwithfellowstudents.Surely,abetterqualitylearningexperiencewouldequatewithahigherlevelofstudentsatisfaction,andwhoisbetterpositionedtomakethatexperiencethebestincanbe,thanstudentsandstaffworkingtogetheronthisdirectly?

5

Basedonthatrealisation–educationisaprocessownedbystudentsandstaff,nottheinstitution–theUniversityofBathandtheStudents’UnionhavetakenaneducationalenhancementapproachtotheNSS.WedecidedthattheNSS–oratleastparticularquestionswithinit–giveusaninsightintohowwellstudentsandstaffengagewitheachotherinrelationtothelearningprocess.ThatapproachallowedustoviewtheNSSasatoolforenhancementandincreasedengagement.Essentially,since2006ourjointapproachhasbeentomaketheinteractionbetweenstaffandstudentsonmattersoflearningas‘reciprocal’asisfeasible.Webelievethataco-ownededucationwouldbeabettereducation,whileallowingfordisciplinedifferences,localeducationalcultures,habitsandpreferences.Forstudentsthismeansanexpectationoftakingresponsibilityforlearning,forgivingfeedbacktostaffandforengaginginenhancementactivity.Forstaffitmeansalisteningear,flexibilityinrelationtodeliveryandinmanycases,themotivatingexperienceofworkingwithamoreengagedstudentaudience.

OurStudents’Unionplaysakeyroleinthispartnershipapproach.IntheearlydaysoftheNSS,ourStudents’Unionsawthesurveyasatoolforenhancementaswell.However,theywerealsodrivenbyastronglyfeltviewthatastrengtheningofthestudentvoicewasimportant.Hence,theSUproposedtorunonlineelectionsforourstudentrepresentatives,allowinganunbiased,fullyelective(asopposedtolocallyselective)approachtofindingrepresentativesforstudentopinion.Alongsidethis,theSUsupportedelectedrepresentativeswithtrainingandongoingbriefingssotheywouldstayabreastofneweducationalorpoliticaldevelopments,bepreparedfortakingeffectivepartinthecommitteestructureandabletodevelopwellunderpinnedproposalsonthosetopicswherestudentswantedtoseechange.Ourstudentrepresentativesarehighlyinformed,haveaccesstothesamedataandinformationthatourstaffhaveandareverypreparedandabletoensureafairrepresentationofstudentopinionisgiven.WehavemanyvisitorstotheUniversity–rangingfrominterestedcolleaguesinotheruniversitiestoQAAreviewers–whohavecommentedontheimpressivepoliticalanddiplomaticcapitalofourstudentrepresentativesandsabbaticalofficers.Needlesstosay,this‘professionalisation’ofthestudentvoicehasgreatlyhelpedtomakestudentsfullpartnersinthestrategicaswellasenhancementprocesses1.

Embracingthepossibilitiestheempoweredandhighlyrepresentativestudentvoicegave,theUniversityrevieweditsarrangementsforstaff-studentliaisoncommitteesatdepartmentallevelanditsformalgovernancestructuremoregenerally.StudentrepresentationonlearningandteachingrelatedcommitteesisnowstandardatalllevelsoftheUniversity.Studentmembersatthehigherlevelsaresabbaticalofficers,facultyrepresentativesorelectedstudentrepresentativeswhoaresupportedbyourStudents’Union’soutstandingRepresentationandResearchManager.Hercontributionrelatesnotonlytosupportingthestudentrepresentativesbeforeandatcommitteemeetings,butalsotosupportcontinuityacrosstheyearsonthemanytopicsunderdiscussion.Thoughsuchahighlevelofengagementwiththeformalcommitteestructureisnodoubtinfluentialinthemidtolongerterm,therealstrengthinrelationtoNSS-measuredstudentsatisfactioncomesfromtheeffectivenessofstaff-studentexchangesatthelevelofthediscipline.Ourstaff-studentliaisoncommitteesareincreasinglychairedbystudentsthemselves,tothegreatsatisfactionofdisciplinestaffacrosstheboard.Allowingsuchanarrangementwasinitiallyfelttobecontroversial,butitappearsthatthelevelofresponsibilitystudentswishtotakefortheirlearningexperienceiscreatinghighlyeffectiveandconstructivediscussionsandresults,notleastinrelationtoavoidingaconsumeristapproachbystudents.Clearly,whenstudentstakeco-responsibilityfortheirstudyexperiencesthenatureofdiscussionchanges,andinourcasemuchforthebetter.

Inpracticalterms,thestaff-studentliaisoncommitteesresolvemanyissuesofinterestorconcerntostudentsatthelevelatwhichtheseissuesoccur.Notallmatters

1 ForaperspectivefromourStudents’Union,seeSection2.1.

6

canberesolvedatthislevelandthisiswhythecommitteeshaveaformalannualfeedbackroutetotheStudents’Union.Byreportingannuallyonissuesthathavebeendiscussed,theStudents’Unionisabletohaveanaccurateandwell-informedoverviewofanyissuesthatneedattentionabovethedisciplinelevelandthathavenotyetbeenraisedduringtheyearthroughothermeans.AsummaryoffindingswithrecommendationsformulatedbytheincomingnewVice-PresidentEducation(aStudents’Unionsabbaticalofficer)isthenreceivedbytheveryfirstUniversity-levellearning,teachingandqualitycommittee,sothataclearagendaforenhancementisset.AgainthisillustrateshowseriouslytheUniversityofBathtakesthewell-informedandco-responsiblestudentvoice.

Beyondthehighlyempoweredstudentvoice,theUniversityitselfofcoursealsotakesactionafterreviewingannualNSSresults.Withineachprogramme,theresultsarescrutinisedandplacedalongsideotherdata,suchasstudentevaluationsofindividualunits,assessmentresults,staffevaluationsandprogrammedata.Staffandstudentshaveworkedtogetheronsettinganenhancementagendafortheprogramme,whichwe’vecentrallysupportedespeciallywhereNSSresultsweredisappointing.Allenhancementactivitiesareundertakenasajointprojectbystudentsandstaff,therebyensuringthatsolutionstoidentifiedproblemsmeettheinterestsofbothparties.Webelievethatthisreciprocalengagementapproachcreatesachangeinthewaystaffandstudentsvieweachother,allowingabetteralignmentbetweenteachingandlearning.Inessence,theNSShelpedstimulateajointinterestinoutstandinglearningandouracademiccommunityhasavoidedthetrappingsofaconsumeristculturewheretheteachingstaffsupplyandthestudentsconsume.

ItisalsotheNSSitselfthatgivesussomedataastowhatextentstudentsfeeltheirdepartmentengageswiththeirfeedback.Theadditionalbankofquestionshasaquestionsetcalled‘Feedbackfromstudents’whichfeaturesthequestion‘Itiscleartomehowstudents’commentsonthecoursehavebeenactedupon’(additionalbankofquestions,QuestionB6.3).TimeandagainweseehowanupwardtrendonthisquestionresultsinastrongeroutcomeonthecoreNSSquestions–especiallyontheacademicaspects.Itappearsthatthemorestudentsknowtheirvoiceisheard,themoretheyratetheteaching,assessmentandoverallsatisfactiontheyexperience.Onfurtherdiscussionwithstudents–wedon’tgobystatisticsalone–wefoundthatasenseofco-ownershipleadstoabetteralignmentofexpectationsandpractices,bothforstaffandstudents.

Ourresultsonthat‘engagementindicator’questionsuggestwearewellabovethetopquartilewithinthecountry,somethingwetakegreatpridein.However,wearebynomeanstheonlyinstitutionthathasundertakenthiskindofinternaldebateandfoundnewwaysofinteractionwiththestudentvoice.UntilrecentlyBirminghamCityUniversityhostedaCentreforExcellenceinTeachingandLearningonpartnershipsandanimportantpartoftheirworkwasonstaff-studentpartnerships.TheUniversityofExeterintroducedtheconceptof‘StudentsasChangeAgents’,whiletheUniversityofLincolndeveloped‘StudentsasProducers’aspartoftheireducationalethos.BasedintheScottishframework,SPARQShashadamajorinfluenceonthesector’sthinkingaboutthestudentvoiceinqualitymechanismsandtheNUSalsohasseveralpublicationsonstudentengagementthatareworthalook.Itappearsthatthestudentvoiceisstrongerthaneverand,undeniably,theNSShasbeenpartoftheprocessofachievingachangeinhowthestudentvoiceisperceived,which–inmyview–helpsalignteachingandlearningmorecloselytoeachother.

What next?

TheNSSwasintroducedin2005.Eightyearson,wemustacceptthatperhapsareconsiderationofthetoolisneeded–evenforenhancementpurposes.Thescoresbecomemoreandmore‘bunched’withinstitutionsnowreachingoverallscoresattheverytopofthescale.Ofcourse,ifthistrulyindicatesthatourstudentbodyacrossthesectorisoutstandinglypleasedwiththeeducationtheyenjoy,thismaybeapositive.However,otherstatisticsavailablewithininstitutionssuchastheStudentBarometer,

7

theStudentOpinionSurveyorhome-grownprogrammeevaluationstelladifferentstory:thereisstillroomforimprovementanddevelopmentinparticularareasthattheNSSdoesnotreach.Moreimportantly,thequestionstillremainswhetherstudentsatisfactionisreallysuchagoodindicatorforahighqualitylearningexperience.Perhapsitisnotsatisfactionthatmatters,butthelevelofactualengagementwithalltheopportunitiesforlearningonoffer.Afterall,evenifstudentsareverysatisfiedwiththelearningexperienceonoffer,itwillonlycreateactualintellectualprogress,ifstudentsactuallyengagewiththoseopportunities.

Withengagementinmind,aninterestingcomparisonisoftenmadewiththeNationalSurveyofStudentEngagementthatisusedintheUS.Thisannualsurvey–secondinsizeonlytotheUScensus–takesaratherdifferentapproachtoevaluatingthestudentexperience.Insteadofconcentratingonwhateducationalofferismadetothestudent,thissurveyturnsthetablesandevaluateshowthestudentsengagewithwhatisoffered.Althoughcontroversialinsomequartersasitaimstomeasurebehaviourasanindicatoroflearning,perhapsthisapproachoffersapossibilityoflookingmorecloselyatthelevelofactualengagementbystudents–bothinthesettingoflearningaswellasthewiderstudentexperience.

Whatwillmatterultimatelyarethechoicesthesectorwishestomake.Takingasector-wideenhancementpointofview,Icannothelpbutwonderwhatwillultimatelystimulateenhancementofthestudentlearningexperiencemore.TheNSSEdatathatcapturehowthestudentsexperiencetheirengagementwithlearningdonotallowforcompetitiveuseinranking,butgiveastrongerinsightintoengagementthantheNSS.Andyet,theNSShasmeantthatuniversities–almostwithoutfail–havenotedtheirstudents’viewsmorethaneverandengagedinsubstantialeffortstoimprovethestudentexperience.WheretheNSSEgivesusthequalityofresponsesthatallowsforenhancement,theNSShasbeenveryeffectiveatmakingteachingandlearningenhancementaninstitutionalprioritysector-wide–perhapsevenmorestronglyso,thananyothereffortsbefore.Maybethefutureliesinthebestofbothworlds:acombinationofdataforenhancementanddatathatcanbeusedasaqualityindicator.Whateverhappensnext,let’shopethatatleastthedebateaboutstudentengagementcontinuesininstitutionsacrossthesector.

Gwen van der VeldenDirector of Learning and Teaching EnhancementUniversity of Bath

GwenvanderVeldenresearchesinstitutionalengagementwiththestudentvoicebasedonherleadershipofthatagendaattheUniversityofBath,herownexperienceofstudents’unionactivismandadvisoryworkwitharangeofuniversities. GwenwasinvitedtowriteonthetopicfortheQAA(‘WhoseEducationisitanyway?’2)andwasamemberofthepanelthatdraftedtherecentlypublishedQAAUKQualityCodeforHigherEducationchapteronstudentengagement.3

2 http://www.qaa.ac.uk/Publications/InformationAndGuidance/Pages/student-engagement.aspx[accessed10October2012]

3 http://www.qaa.ac.uk/publications/informationandguidance/pages/quality-code-B5.aspx[accessed10October2012]

8

1. Introduction

TheNSSisapowerfulforceinUKhighereducation.Ithasbeenpraisedforhelpingtoopenuptheworldofundergraduatestudytoscrutinyanddebate.Ithasbeenthetargetofheartfeltcriticismfromthoseopposedtotheideaofstudentsasconsumersandtheriseofthemarketinhighereducation.ItisnowakeypartofthecurrentGovernment’splantousestudentchoiceasamorepowerfulleverforqualityimprovement.Butaboveall,theNSShasattractedattention.Mostofthatattentionhasfocusedontheroleofthesurveyintheaccountabilityofuniversities,andinprovidinginformationtoprospectivestudents.WhatistypicallyoverlookedinthispolariseddebateisthepartthattheNSScanplayininstitutionalprocessesfortheenhancementoflearningandteaching.

TheuseoftheNSSforenhancementisbecomingincreasinglycommonwithininstitutionsthemselves,butthereisalackofsharedpracticeanddiscussionabouthowthatcanbeachieved.Thisreport,basedontheexperiencesofarangeofinstitutions,isdesignedtoprovideanoverviewoftheroleoftheNSSinenhancement,alongsidespecificexamplesofinstitutionalpractice.Itisalsointendedtoraiseawarenessofthisaspectofthesurvey,andtoprovokedebateabouttheopportunitiesandobstaclesofusingitinthisway.

TheNSSdidnotoriginateasatoolforenhancement.Theprimaryintentionwastocreateastudentsurveythatwoulda)yieldinformationthatcouldbeusedbyprospectivestudentstochoosesuitablecourses,andb)providealight-touchformofqualityassuranceandpublicaccountability4.WiththerecentchangestotheUKHEsystem(mostnotablyinEngland)thefirstofthoseroleshasbecomeparticularlyprominent,buteversinceitsintroductionin2005theNSSresults–throughreportinginthemediaanduseinleaguetables–havehadaprofoundimpactonhowinstitutionsarepresentedtothepublic.

Thatdevelopmenthasforcedinstitutionstopayattentiontostudents’perceptionsoftheirlearningenvironment.TheNSShasahighlevelofvisibilitywithininstitutions,andthereareoftensophisticatedmethodsforinternaldisseminationofthedatatofaculties,departmentsandprogrammeleaders.Thesurveyresultsoftenalsoplayakeyroleindecision-makingprocessesasanimportantsourceofmanagementinformation.[Section 3.2 Integration into decision-making]

ThesimplisticuseoftheNSSscoresinthemediacanleadtoanunderstandablesuspicionandresentmentonthepartofacademicstaff,especiallyiftheyperceivethisperspectiveintheattitudeofinstitutionalmanagers.Thosemanagers’legitimateconcernabouttheoutward-facingaspectsoftheNSS(suchasparticipationratesandcomparisonsbetweenuniversitiesandcourses)canthenbeinterpretedby‘frontline’staffasasimplisticpressuretoraisescoreswithoutconcernforgenuineimprovementsinlearning.[Section3.1 Staff attitudes]

However,concomitantwiththekindofdeficitmodelinspiredbyleaguetables,theNSSbringsanincreasedfocusonlearningandteaching,thatforsomeinstitutionscanbebothnovelandwelcome.Whenusedwell,itcanbeapowerfulleverfor

4 HEFCE(2004)NationalStudentSurvey2005:Outcomesofconsultationandguidanceonnextsteps.Bristol:HEFCE.Availablefrom:http://webarchive.nationalarchives.gov.uk/20120118171947/http://www.hefce.ac.uk/pubs/hefce/2004/04_33/[accessed11October2012]

9

change,inspiringconversationsaboutlearningandteaching,andengagingstaffintheenhancementprocess.[Section3.3 Impetus for change]

ThecontributionstothisreportstronglysuggestthatwhiletheprominenceoftheNSSwasinitiallyinspiredbyitsimpactoninstitutionalreputation,recruitmentandfinances,thefocushasstartedtoshifttowardshowitcansupportthegenuineimprovementoflearningandteaching.TheuseoftheNSSforenhancementmaybeaby-productofitsreputationalimpactbutitneverthelesshasthepotentialtobeanimportanttoolforimprovingpractice.

UsingtheNSSinthiswayisnotwithoutitscomplications.Thefactthatitwasnotprimarilydesignedwithenhancementinmindcausesproblems,asdoesthewayinwhichthemediauseitasasummativemeasureofteachingquality.Thefocusthatitcreatesonstudents’satisfactionwithwhattheyhavereceived,ratherthanthelevelandqualityoftheefforttheyinvest,isalsoachallenge.[Section4.1 Limitations and challenges]

Butdespitetheseobstacles,institutionshavefoundinnovativeandinterestingwaysofdeployingiteffectivelywithintheenhancementprocess.ProbablythemostimportantelementofthisistheacknowledgementthattheNSSisonlyapartialpictureofstudents’experiences,andworksbestasastarting-pointforfurtherinvestigation.Itcanindicateareasofpossibleconcernorsuccess,butitwillnever,usedinisolation,yieldadefinitiveunderstandingofeducationalqualityorthelevelofstudentlearning.[Section4.2 Triangulation with other data]

Thesimplisticcomparisonsmadebythemediamaybeunhelpful,buttheappropriateuseofbenchmarkingisavitalaspectofusingtheNSS.Comparingscoreswithcomparatorcoursesorinstitutionscanbeausefulstepinidentifyingareasforimprovementandgoodpractice,providedthatdifferencesincontextaretakenintoaccount.[Section4.3 Benchmarking]

ThebenchmarkingandanalysisofthenumericalNSSscoresprovidesagoodfirststep,butsomeinstitutionshavefoundsuccessinusingthestudentcommentsfromthesurveytogainricherinformation.Focusgroupsareanotherusefulsourceofqualitativedata,allowingissuestobeexplored,andpossiblesolutionstobeconsidered.[Section4.4 Using qualitative data]

TheuseoffocusgroupsisonewayofinvolvingstudentsintheexplorationofNSSresults,butastrikingfeatureofhowenhancementstrategieshavedevelopedoverrecentyearsisthattheinvolvementofstudentsintheNSSstretchesfarbeyondsimplybeingsourcesandobjectsofdata.Therearemanyexamples,includingsomeinthisreport,oftheNSSprovidinganextraspurandausefulfocusforstaff-studentpartnerships,andbeingusedinapositivewaytoputthestudentvoiceattheheartoflearningandteachingcultures.

ConnectingtheuseoftheNSSwiththewiderprocessofstudentrepresentationhasbeenimportanttomanyinstitutions.Byworkinginpartnershipwiththestudents’union,andbyencouragingandsupportingcourseandfacultyrepresentativestoparticipateinthediscussionofNSSresults,asenseofsharedownershiphasbeenfostered.[Section2.1 Student representation]

Institutionshavealsoinvolvedstudentsintheanalysisandexplorationoftheresults,throughsupportingthemasresearchers,orthroughmoreinformalmeanssuchasconferencesandworkshops.[Section2.2 Student exploration]

Andalotofefforthasbeeninvestedin‘closingthefeedbackloop’,reportingbackontheactionsthathavebeentakeninresponsetotheNSS.Buildingthesurveyintoagenuinedialoguebetweenstaffandstudentsdependsonstudentfeedbackbeingvisiblyacknowledged,justasitrequiresinstitutionstoavoidtreatingNSSresultssimplyasdemands.[Section2.3 Closing the loop]

10

TheNSSisinfluentialandcontroversial.Boththesurveyitselfanditsimpactonthesectorhaveattractedcriticism,bynomeansallundeserved.Nevertheless,moreconversationshavetakenplace–ininstitutions,facultiesanddepartments–aboutlearningandteachingenhancementasaresultofthesurvey.Andtherearesignsthatthesurveycanbeusedinawaythatdoesnotstrengthenadamagingconsumerism,butinfactpromotesideasofstudentempowerment,involvementandrepresentation.ThisreportisanattempttorecognisethatcontributionoftheNSStoenhancement,andtohelpsupportandstructuretheconversationswhichareinspiredbytheNSS,butareultimatelyconcernedwithimprovingstudentlearning.

1.1 About this report

ThisreportisbasedontheexperiencesoftheHigherEducationAcademy’sNSSInstitutionalWorkingGroup,madeupofindividualswhoworkcloselywiththeNSSincentralroleswithinhighereducationinstitutions.Thegroupmeettwice-yearlytodiscussandsharewaysofusingtheNSSforenhancement(seebelowformoredetail).Themaintextisbasedprimarilyonresponsesofgroupmemberstoashortsetofopen-endedquestions,andfollow-upconversations.ThosequestionsfocusedonhowtheNSSisperceivedandused,howstudentsareinvolved,andtherespondents’ownthoughtsaboutitsbenefitsanddrawbacks.Thestructureofthereportisbasedonthethemesthatemerged.Inadditiontothesewrittenresponses,thereportalsodrawsonnotes,conversationsandpresentationsfrommeetingsofthegroup.

Themaintextofthereporthasbeenanonymised,toenableindividualstobehonestandopenbothabouttheirinstitutionalpractices,andtheirpersonalviewsoftheNSS.Quotationsfromthewrittenresponsesareusedwithinthereport(indentedanditalicised)andthesehave,ifnecessary,beenparaphrasedinordertopreventidentificationofindividualsorinstitutions.Thereportalsocontains9shortcasestudies,contributedbymembersofthegroup,thatprovidemoredetailed,specificandnon-anonymousexamplesofinstitutionalpractice.

Thereportisdividedintothreemainchapters,lookingatStaff-studentpartnerships;Institutionalstructures;andAnalysisandexploration.Thechapterseachconsistofseveralsectionswhichinclude,alongsidethemaintextandcasestudies,keypointsandquestionstopromptfurtherreflection.Thekeypointsandquestionsarecollectedtogetherattheendas,respectively,SuggestionsforgoodpracticeandQuestionsforreflection.

Thereportisnotdesignedtoprovideanswers,buttoofferideas,examplesandinspirationforpractice.BydistillingtheexperiencesandreflectionsofagroupofpeoplewhoarecloselyengagedwiththeNSSwithinarangeofinstitutions,wehopeitwillencourageandenableotherstoreflectontheirownuseofthesurveyfortheenhancementoflearningandteaching.ItisimportanttonotethatthereportdoesnotgivetheHEA’sviewoftheNSS,norshoulditbeassumedtorepresenttheviewofanyindividualmemberofthegroup,ortheirinstitution.

Whatdoesthisreportdo?

• Itsuppliesanoverviewofinstitutionalpractice,coveringthebroadrangeofissuesthatarisewhenusingtheNSSaspartoftheenhancementprocess.

• Throughtheshortcasestudiesincludedineachsection,itprovidesarangeofspecificexamplesofhownamedinstitutionshaveusedtheNSS.

• ItdescribesthechallengesanddifficultiesoftryingtousetheNSSasanenhancementtool,withexamplesofhowinstitutionshaveattemptedtoovercomethem.

• Itincludessuggestionsforinstitutionstoconsiderintheirownactivities.

• Itpromotescriticalreflection,throughpromptsandquestionsdesignedtofosterdebateanddiscussionoftheroleandpotentialoftheNSS.

11

Whatdoesthisreportnotdo?

• Itdoesnotgivespecificadviceonstatisticaltechniquesormethodsofdatareporting.

• Itdoesnotattempttosupplyinterpretationsofloworhighscoresforparticularquestionsorscales.

• ItdoesnotprovideideasforsimplyimprovingNSSscores,beyondthesuggestionsformakinggenuineimprovementstolearningandteaching.

Whoisthisreportfor?

• StaffworkingdirectlywithNSSdata,ordatafromotherstudentsurveys,whetherthroughanalysis,disseminationoractionplanning.

• StaffinterestedinhowtheNSScouldbeusefulforinformingenhancementstolearningandteaching.

• SeniormanagerswantingtomakethemostoftheNSSinplanningandreviewprocesses.

• Althoughthereportisdrawnfrom,andaimedat,staffwithininstitutions,itmaybeusefultostudentsandstudents’unionswantingtoengagewiththeNSSasawayofunderstandingandrespondingtostudentfeedback.

Thereportisintendedtobeusedalongsideothersourcesofinformation.Section5.3describesasmallnumberofusefulresearchpublications,andotherresourcesproducedbytheHEA(includingdataanalysis)arelistedinSection5.4.

1.2 About the Institutional Working Group

TheNSSInstitutionalWorkingGroupmeetstwiceeachyeartosharepracticearoundusingtheNSSforenhancement,andtodiscusscommonproblemsandchallenges.ThegroupishostedandsupportedbytheHEA.ThemembersofthegroupusetheNSSinarangeofcapacities,buttheymostlyworkwiththesurveyataninstitutionallevel,inlearningandteachingunitsorrolesrelatedtoeducationalquality.

Meetingsfocusondiscussionsandworkshops,andonpresentationsfromgroupmembershighlightinginterestingandinnovativeaspectsofinstitutionalactivities.Thegroupoffersaspaceforfreeandfrankconversation,inthat–likethemaintextofthisreport–nocommentsmadewithinmeetingsareattributedtoindividuals.ItisavaluablespacewherepeopleworkingcloselywiththeNSS,andattheforefrontofusingitforenhancement,canmeetandcollaborateonanissuethatisoftenattheheartofcompetitionbetweeninstitutions.Thegroupregularlypublishescasestudies,suchasthosecontainedinthisreport.Earliercasestudiesareavailableatthefollowinglocations:

2007casestudies:http://www.heacademy.ac.uk/assets/documents/subjects/bioscience/nss-case-studies.doc2010casestudies:http://www.heacademy.ac.uk/assets/EvidenceNet/Case_studies/NSS_case_studies_Nov_2010.pdfCurrentandrecentmembersofthegroup:

UniversityofBath–ShaunMcGall(LearningandTeachingDevelopmentOfficer)BishopGrossetesteUniversityCollegeLincoln–VickyMays(PolicyOfficer)CardiffMetropolitanUniversity–NicolaPoole(StudentRetentionOfficer)CoventryUniversity–AndrewTurner(TeachingandLearningProgrammeManager)

12

UniversityofDerby–SueMorrison(SeniorAssistantRegistrar,StudentExperience)DurhamUniversity–RichardHarrison(HeadoftheAcademicSupportOffice)UniversityofGlamorgan–DenizeMcIntyre(SeniorProjectsOfficer)Goldsmiths,UniversityofLondon–ClareBlake(QualityAdministrator)UniversityofHertfordshire–CatherineRendell(DeputyDirectorofAcademicQualityAssurance)UniversityofHuddersfield–KathySherlock(HeadofRegistry)LancasterUniversity–AdamChild(AssistantRegistrar)UniversityofLeeds–JennyLyon(AssistantRegistrar)LeedsMetropolitanUniversity–AlisonJones(HeadofStrategicPlanning)LiverpoolJohnMooresUniversity–ClareMilsom(HeadofAcademicPractice)Queen’sUniversity,Belfast–NualaToman(EducationalDeveloper)SheffieldHallamUniversity–JasonLeman(SeniorLecturerinResearchandEvaluation)SouthamptonSolentUniversity–ChrisRapley(SeniorResearchOfficer)UniversityofStirling–BobMatthew(Director,CentreforAcademicPracticeandLearning)UniversityofSunderland–BeatriceOllerenshaw(DirectorofAcademicServices)UniversityofWinchester–YazEl-Hakim(DirectorofLearningandTeaching)UniversityofWorcester–JulianMartin(HeadofQualityAssuranceandEnhancement)

1.3 About the author

DrAlexBuckleyworksintheStudentSurveysteamattheHEA,andleadsontheiractivitiessupportinginstitutionstousetheNSSforenhancement.SincejoiningtheHEAinJanuary2010,hehasfacilitatedtheInstitutionalWorkingGroupaspartofhisrole.Hisworkalsoincludesdatareporting,andresearchonstudentsurveys,studentengagementandtheconnectionsbetweenthetwo.BeforejoiningtheHEA,AlextaughtappliedandprofessionalethicsattheUniversityofLeeds,aftergainingaPhDinphilosophyfromthesameinstitutionin2008.

1.4 Acknowledgements

Firstofall,IwouldliketothankallthemembersoftheInstitutionalWorkingGroupfortheiractivecontributionsatmeetings.Thosemeetingshaveformedthebackgroundtothisreport,andthemeetingnotesandpresentationshaveprovidedmaterialforthetext.Iwouldadditionallyliketothankthosemembersofthegroupwhosubmittedwrittenresponsestothesurvey:ClareBlake,ClareMilsom,ChrisRapley,JasonLeman,JennyLyon,RichardHarrison,NicolaPoole,AlisonJones,BeatriceOllerenshaw,NualaToman,AdamChild,DenizeMcIntyre,BobMatthew,KathySherlock,YazEl-Hakim,JulianMartinandCatherineRendell.Thoseresponsesweredetailed,oftenlengthy,andprovidedhonestassessmentsoftheNSSasanenhancementtool.Extrathanksgoestothosememberswhoalsocontributedshortcase-studies.Andwearealsogratefultotherepresentativesofstudents’unionswhocontributedtheirperspectivesatshortnotice:AlexPool,Vice-President(Education)attheUniversityofBathStudents’Union,RuthFoster,PresidentoftheStudents’UnionatCardiffMetropolitanUniversityandKateWilkinson,Vice-President(AcademicQuality)atLiverpoolStudents’Union.IamverygratefultoGwenvanderVeldenforwritingtheforewordtothereport,sharingbothherviewpointontheNSSandtheinspiringdevelopmentsatBath.Finally,thankstoPaulBennett,JasonLeman,CatherineRendell,ClareMilsom,NicolaPooleandAnnaHighmoreforcommentsonearlierdraftsofthereport.

13

2. Staff-student partnerships 2.1 Student representation

Key points

• NSSresultsneedtobeconnectedtothewiderprocessofstudentrepresentation,toensurethatthesurveyispartofthewiderconversationsbetweeninstitutionsandstudents.

• IfstudentsaregoingtoplayanactivepartinthediscussionofNSSresultsincommitteesandworkinggroups,thentheyneedsupport,trainingandfullaccesstotherelevantresourcesanddocuments.

• Partnershipsbetweeninstitutionsandstudents’unionsareoneofmostimportantelementsofinvolvingstudentsintheNSS,butboththeopportunitiesandthechallengesshouldbeopenlyacknowledgedanddiscussed.

MostinstitutionsfeelstronglythattheNSShasincreasedthevisibilityandimpactofthe‘studentvoice’.ThereputationaleffectoftheNSSmeansthatinstitutionsarenowimpelledtogivegreaterconsiderationtotheirstudents’opinions,butsomeareclearthatthishashadagenuinelypositiveimpactonstaff-studentrelationships.

The work involved in the promotion of the survey, the analysis and dissemination of results, and the reacting and responding to results has generated a strong collaborative approach … This sense of shared responsibility and ownership has enhanced how the institution implements change more generally.

FortheNSStobepartoftheconversationbetweenstudentsandtheinstitutionratherthansimplyasetofdemands,itisimportantthatthesurveyisintegratedintothesystemsofstudentrepresentation.ApowerfulmessagefromseveralinstitutionsisthattheuseoftheNSSasatoolforenhancementdependsontheextenttowhichithelpstobuildproductivestaff-studentpartnerships.

Oneofthekeyaspectsofstudentrepresentationistheinvolvementofstudentsincommitteesandworkinggroups,atthelevelofmodules,coursesanddepartmentsrightthroughtoinstitutional-levelcommitteesandsenates.AsNSSdatausuallytraceayearlycoursethroughthosecommittees,animportantmechanismforconnectingtheNSStostudentrepresentationmorewidelyistheinclusionofstudentsinthatprocess.Thenextsectionlooksathowstudentscanbeinvolvedintheanalysisandinvestigationofthedatathemselves;thissectionisfocusedonhowstudentsareinvolvedinhowthedataarerespondedto.

Asthenamesuggests,thiskindofstudentinvolvementismostcommoninstaff-studentliaisoncommittees,butthereareotherwaysinwhichstudentsareincluded:

• manyinstitutionsincludestudentrepresentationonfacultyandinstitutionallearningandteachingcommittees;

• oneinstitutionreportedthatgroupscontainingstudentsareoftensetupspecificallytodiscussstudentfeedback,oftenfocusingontheNSS;

14

• someinstitutionshavefacultyforumswhereseniorstaffandstudentrepresentativesmeetanddiscussnewlyreleasedNSSresults;

• oneinstitutionhasannualforumstoallowseniormanagersandstudents’union(SU)5officerstodiscusstheresults.Theseforumsrewardhigh-performingareas,identifyprioritiesforenhancement,andcreateactionplans;

• anotherinstitutioncreatesad hocworkinggroupstodealwithspecificissuesthathavearisenfromNSSresults.

Thestudentmembersofthesecommittees,groupsandforumsneedtobeactivelyincludediftheirpresenceistobemeaningful,whichrequiresthattheyareprovidedwiththeresourcesandsupportthattheyneed,andaretreatedasgenuinepartnersintheconversation.

SeveralinstitutionsensurethatstudentrepresentativesonthesegroupshaveaccesstotheNSSresults,andthisisalsoapriorityformanySUs(Section2.2looksinmoredetailathowinstitutionshavesupportedstudentstoundertakeanalysisofthedata).Inrelationtoothersupport,someinstitutionsprovidetrainingtostudentrepresentatives,aswellasfacilitatingaccesstoothersourcesofdatatoallowthemtocontextualisetheNSSresultsforthemselves.Atanationallevel,theNationalUnionofStudents(NUS)providesusefulsupportandresources6,andSPARQSinScotlandarefocusedonprovidingtrainingandsupportforstudentrepresentatives7.

Acknowledgingthetimeinvestmentthateffectivestudentrepresentationcaninvolve,severalinstitutionshavecreatedpaidpart-timerepresentativepoststoprovideaconnectionbetweentheinstitution(anditsconstituentschoolsanddepartments),theSUandthestudentbody.Theseposts,explicitlymodelledinonecaseontheNUS’‘NSSAmbassadors’8cancoverarangeofactivities,includingcollatinginformalfeedbackfrompeers,promotingtheNSSandothersurveys,andactingasrepresentativesofstudentgroups.

Case study: The role of students in quality assurance and enhancement at the University of HuddersfieldTheUniversityvaluesthestudentvoiceasanimportantelementofqualityassuranceandenhancement.StudentsarerepresentedonUniversity-widecommittees,forexampleCouncil,Senate,UniversityTeachingandLearningCommittee(UTLC),AnnualCourseEvaluationandatSchoollevelonSchoolBoards,CourseCommitteesandStudentPanels.InadditionSabbaticalOfficersarepanelmembersonStudentAppeals,Complaints,DisciplinariesandFitnesstoPracticeCommittees.InrelationtotheNSStheresultsareanalysedbytheDeputyVice-ChancellorandPro-Vice-Chancellor(TeachingandLearning)beforebeingputintoamoreunderstandableformatbythePlanningandInformationServicetobediscussedatUniversityandSchoollevels.TheresultsaremadeavailabletotheStudents’Uniontoallowthemtoengageindiscussiononresponses.

5 ThestudentrepresentationorganisationsinsomeScottishinstitutionsarecalled‘students’associations’.Inthisreporttheexpression‘students’union’isintendedtocovertheseorganisations.

6 http://www.nusconnect.org.uk/campaigns/highereducation/nss/[accessed6October2012].7 http://www.sparqs.ac.uk/[accessed6October2012].8 http://www.nusconnect.org.uk/campaigns/highereducation/nss/ambassadors/[accessed6October2012].

15

TheQualityandStandardsAdvisoryGroup(QSAG)isthe‘workhorse’ofUTLC;itischairedbythePro-Vice-Chancellor(TeachingandLearning)andmembershipincludesstaffdrawnfromSchools,ServicesandtheStudents’Union.AtQSAG’sfirstmeetingoftheyearNSSresultsarereviewedandthoseSchoolswhoseresultsarebelowacertainlevelofstudentsatisfactionareaskedtoprovideactionsplansaddressingtheissues,andthesehavetobeavailableforthenextmeeting.SchoolactionplansaremonitoredbyQSAGandtheUniversity’sSeniorManagementTeam.

In2008-09theUniversityimplementedstudentrepresentationonThematicReviewsandReviewsandRevalidations.Thelatterhadbeenrevisedfromtwoseparateprocessesintooneactivity.Thepanelshadalwaysmetwithstudentsaspartofthereviews;however,nowmembersoftheStudents’UnionExecutiveareincludedonthepanelandinvolvedindiscussionsaboutthestudentexperienceandNSSoutcomes.ReviewscantakeuptotwodaysoutofaworkingweekandthisisthemainreasonwhystudentsarerepresentedherebytheSabbaticalOfficers.AThematicReviewonthesubjectofassessmentandfeedbackledtoanHEAChangeAcademywiththeSUPresidentasamemberoftheproject.Thebenefitsoftheprojecthavebeendemonstratedyearonyearwithimprovedlevelsofstudentsatisfactioninassessmentandfeedback,whicharenowsomeofthebestintheUK.

TheUniversityrecognisesalloftheabovecantakeupvaluablefreetimeofthestudentandhasworkedwiththeSUdevelopingmechanismstoensurestudentsaresupportedwithappropriatetrainingandongoingsupport.Inrecognitionofthecontributionmadebystudents,theSUandtheUniversity’sCareersServicedevelopedtheStudentTrainingandRecognitionScheme(STARS)providingstudentswithastructuredmethodtorecordandreflectontheirexperiences,enhancingtheirCVandfutureemploymentprospectsandgainingagradedrecognitionofthisawardedjointlybytheUniversityandtheSU.

UniversitiesshouldnotunderestimatethebenefitsofworkingcollaborativelywiththeSU.AttimesinthepastitwasnotalwayseasytotrackdownwhoshouldbeattendingvariouscommitteesandespeciallyatthestartoftheyearwhentherewasanewSUExecutiveandnewstudentrepresentatives.In2009,however,theSUappointedafull-timeDemocraticandStudentRepresentationCo-ordinatorandwehavedefinitelybenefitedfromthisappointment.AlongsidethebespoketrainingandcommitteestructurewejointlyorganisetheannualStudentVOICEconference,withapproximately60studentrepresentativesandtheSeniorManagementoftheUniversityactingasfacilitators,andthe‘AskTim’sessions,wherestudentsmeetwiththePro-Vice-Chancellor(TeachingandLearning–‘Tim’)inopenmeetingsonfiveorsixoccasionsthroughouttheyear.

Kathy SherlockUniversity of Huddersfield

Theprimaryintendedbenefitofstudentstakinganactivepartinthesegroupsandcommitteesistoensurethatthedecisionstakeninthosemeetingsareinformedbystudents’perspectives,andhavesomeelementofbuy-infromthestudentbody.Thisrequiresanenvironmentinwhichstudentrepresentativescanvoicetheiropinionsandexperiences,engageinopendiscussion,andhavegenuineinputintotheoutput,whateveritmaybe.Ensuringthatstudentsaretreatedasgenuinepartnersinthesemeetingsisthereforeasimportantastrainingandaccesstodata.Oneinstitutionhastriedtoachievethatthroughanumberofmethods.

16

• theyhavegivenstudentsaccesstoallpapersbeforethemeeting;

• theyhaveincludedstudentsinthecirculationofallminutes,plansandreportsthatresultfromthemeetings;

• theyhavemadesurethattherearenoconfidentialitemsontheagenda.

Student representatives are present on all groups and are given access to the data reports and the action plans – there is no ‘closed’ business.

Anotherinstitutionwascarefultoensurethattheissuesthatstudentrepresentativesmightbemorekeentodiscusswerenotplacedattheendofagendas,orburiedinlongmeetingsamidbureaucraticandproceduralitems.

Itisalsoimportantthatinstitutionsacknowledgeboththeacademicprocessesandterminologywithwhichstudentsareunlikelytobefamiliar,andthepowerrelationshipsthatmayexist.StudentrepresentativesmaywellbecommentingonNSSresultsforcoursesthattheythemselvesaretaking,indiscussionswithacademicswhoarebothteachingthem,andassessingtheirwork.Itisimportantthatthisissueisacknowledgedandapproachedwithsensitivitybybothstudentsandstaff,ifstudentparticipantsaretofulfiltheirjobofrepresentingtheviewsofstudentgroups.

Thereareexamplesofinstitutionsinvolvingstudentsbeyondthemeetingsthemselves:

• oneinstitutionaskedcourserepresentativestoreadandcommentondepartmentalactionplans.Thisproducedunexpectedoutcomes,withthestudentsexpressingconcernthattheactionplanswereunrealistic,andthatthismayhaveademoralisingeffectonstaff.

• anotherinstitutiontaskedtheSUandanacademicmemberofstafftoco-produceasummaryreportbasedontheannualreportsoftheindividualstaff-studentliaisoncommitteesaroundtheinstitution.Thisreportcontainedrecommendationsfortheuniversitylearningandteachingcommittee,whichwerethenturnedintotheinstitutionalactionplan.

SeveralinstitutionshighlightedtheroleoftheSUinthiskindofsupportfor,andengagementof,studentrepresentatives.SUshavealevelofreach,expertiseandenthusiasmthatmaketheminvaluablepartnersininvolvingstudentsintheprocessofusingtheNSS.Mostoftheinstitutionalexamplesofsuccessfulattemptstoinvolvestudentshaveinvolvedco-operationwiththeSU.

One of the benefits of the NSS is that it provides a context for improved collaboration within the University and between the University and the Students’ Union and the wider student body.

17

The view from a students’ union – University of BathTheNSSisaworthwhiletoolthatenablesadialoguebetweentheUniversityandtheStudents’Union.Likeanysurveyithasitslimitations,butbycreatingthatdiscussionitcanbeincrediblybeneficialforbothpartiesconcerned.TheNSShostsarangeofquestionswhichcanthenbeusedtobenchmarknotjustexternallywithotherinstitutionsbutalsointernallybetweendepartments.FromthatprocesswecanlookatidentifyinggoodpracticeandrollingthatoutUniversitywide–thatcanonlytakeplacethroughstudentfeedbackandconsultation.Studentsaretheonlyoneswhoareexpertsinwhatitisactuallyliketobeastudent,thereforeifadepartmentoraparticularquestionontheNSSisscoringlowerthantheothersitisimportantthatinsteadofhypothesisingthereasonsstudentsarejustasked.ThereforeproposalscanbeidentifiedandtogethertheUniversityandStudents’UnioncanworktogethertomaketheStudentExperienceatBathstronger.LikewiseifonedepartmentisexcellingatanareathenwecansharebestpracticeacrosstheUniversity.

Weidentifybestpracticethroughanumberofareas,whetherit’sAcademicRepssittingonStaff-StudentLiaisonCommittees,FacultyRepsonFacultyLearningandTeachingCommitteesorStudents’UnionOfficershavingone-to-onemeetingswithUniversitystaff.AtBathwehaveanincrediblystrongsystemofengagingwithreps–wehaveAcademicCouncileverythreeweekswherealloftheAcademicRepsfromacrosstheUniversitycometogethertodiscussissues.TheNSSgivesagoodsteerofareastoinvestigateanddevelopfortheyearahead.OnlybytheUniversityandStudents’Unionworkingtogethercanissuesproperlybeidentified,bestpracticeidentifiedandshared,andthegeneralStudentExperiencestrengthened.

Alexander PoolVice-President (Education)University of Bath Students’ Union

WorkingwiththeSUdoesofcoursecreatechallenges.Twoinstitutionsdescribedthetensionbetweentheneedtobuildupstableandlong-termrelationshipswiththeSU,andtheshort-termelectedstatusofstudents’unionofficers.Withtermslastingoneortwoyears,andprioritiesandattitudesoftenchangingwitheachnewpostholder,itcanbechallengingtocreatelastingplans.Thisriskcanbemitigatedbyensuringthatprojectsinvolvethepaidstudents’unionstaff,whoareusuallyaroundforlonger,alongsidetheelectedofficers.Thisisfelttoprovideabasisofcontinuitythatcanhelptocreatesustainableprojectsandpartnerships.

SUscandifferintheirlevelofengagementwiththestudentbody.OneinstitutionexpressedtheirconcernabouttheSU’sengagementwithparticulargroupssuchasethnicminorities,part-timestudentsandpostgraduates.Forsometypesofstudents,theSUmaynotbeashort-cuttoengagement,andinstitutionsandSUsmayhavetoworktogethertoinvolvethosestudentsinthekindsofactivitiesdescribedinthissection.

Staff-student interactions around change have significantly increased. Some of this may well have happened without the NSS, but it has been a major lever for change.

ThesearearangeofwaysinwhichstudentshavebeeninvolvedindiscussingtheresponsetoNSSresults,primarilythroughinstitutionalcommitteestructures.Thosecommitteestructuresmaybenecessarilycomplex,butwithineachindividualmeetingorforumtheissueissimple:tocreateanenvironmentwherebothstaffandstudentsareabletoprovidepersonalinput,intheformofexperiences,viewsandsuggestions,onhowstudentfeedbackshouldberespondedto.InthenextsectionwelookinmoredetailathowstudentscanbeinvolvedinanalysingandinvestigatingtheNSSdata.

18

Questions for reflection:

• ArestudentspresentonallthecommitteesandgroupswhereNSSresultsarediscussedinanydetail?

• Arestudentrepresentativessufficientlysupportedtoallowthemtoplayanactiveroleinmeetings?

• DoestheinstitutionandtheSUworktogetherontheNSS,inawaythatishonest,productive,sustainableandacknowledgesthechallengesthatexist?

19

2.2 Student exploration

Key points

• Withsupportandadvice,studentscanplayanactiveroleintheanalysisofNSSdata,whetheraspaidresearchers,throughtheSUorthroughtheirstudies.

• Jointstaff-studentevents,suchasworkshopsandconferences,canprovideaninformalbuteffectivemethodofinvolvingstudentsintheexplorationofNSSresults.

• SUsareentitledtoreceivefullaccesstothedata,whichcreatesexcellentopportunitiesforstaff-studentcollaborationsondataanalysisandinvestigation.

NSSdatadonotarriveaccompaniedbytheirinterpretations.Theyneedtobeunpacked,discussedandcontextualisedinordertobeusefulasguidestothelearningexperiencesofstudents,andasfoundationsforqualityenhancement.Chapter4ofthisreportisfocusedoninstitutionalprocessesforanalysingandsupplementingthedata.Section2.1lookedathowstaffandstudentscanworktogethertorespondtostudentfeedback.Thissectionisfocusedonsomethingrelatedtobothofthese:howstudentscanbeinvolvedinexploringandinvestigatingtheNSSdatathemselves.

OneofthemostcommonwaysofinvolvingstudentsinNSSexplorationisasparticipantsinfocusgroups.ThisisdiscussedinSection4.4,whilethissectionisfocusedonhowstudentscanplayanactiveroleasresearchersandpartnersintheprocessofexploration.

The view from a Students’ Union – Liverpool John Moores University

Asyouwouldexpect,westartwithananalysisofstudents’satisfactionwiththeircourseineachofthekeyNSSquestionareas,includingoverallsatisfaction.Wethentakeatargetedapproachtoeachofthesecourses,especiallythosewiththeleastsatisfiedstudents.

ThisyearthistargetedapproachmeansthatwefirstlytalktotheProgrammeLeaderabouttheircourse,whatprioritiestheyhavefortheyearahead,andwheretheythinktheycanimprovestudentsatisfaction.WealsodiscusswiththemtheNSSqualitativecomments,meaningthatbythefirstfewweeksoftheacademicyearweknowthekeythemesthatweneedtoraisewithstudentsandCourseRepstoseestudentsatisfactionrise.

Wethenstartgoingoutandtalkingtothesestudentsaroundcampus,normallywewillknowwhenandwheretheirbiggestlectureissothatwecantargetthem.WealsoprovideprioritisedsupportfromtheCourseRepsonthesecourses,sothatwecanbesurethattheyaredoingtheirjobproperly,theyfeelsupported,andtheyarehavingimpact.Occasionally,ifthereisaspecificissuethataProgrammeLeaderoraCourseRepwishestoaddress,wewillfacilitateafocusgrouponthatissueandsendrecommendationstotherelevantpeopleinvolved.Allthisworkisthenwrittenupintoa“FacultyStudentVoiceReport”foreachofourfaculties,andsenttotherelevantDean.

SincetheinsertionofthenewNSSquestion23onstudentsatisfactionwiththeirstudents’unionwehavealsolookedtoenhancethestudentexperienceofthosecourseswhoareleastsatisfiedwithourselves.Thismeansthatwehavetargetedthesecoursesforacademicsocieties(iftheydon’thavethemalready),tripsthatcouldenhancetheiracademicexperienceandalsospecificcommunicationswiththosestudentstoensurethattheyknowwhatwedoforthem.Wealsoanalysethesestudents’satisfactionwiththeuniversity,sothatiftheyaredissatisfiedwithbothinstitutionswecanworkwiththesestudentstosecureimprovedsatisfactionforbothparties.

Kate WilkinsonVice-President (Academic Quality)Liverpool Students’ Union

20

SUshaveverysimilaraccesstotheNSSdataasinstitutions,throughtheNSSResultsWebsite.Whilethisdoesnotprovideaccesstoindividualresponses,itdoesprovideaccesstoaggregatedresultsforgroupsassmallastenresponses.Thisismoredetailedthanthepubliclyreleasedresults,andthisaccesstorelativelyfine-graineddatameansthattherearesignificantopportunitiesforSUstoengageinanalysisandexploration,opportunitiesthatSUsseemtobeoftentakingup.

The Students’ Union gets a full copy of all the NSS data – they do an analysis for themselves.

Onthepartoftheinstitution,helpcanbeprovidedintheformoftrainingandexpertisesharedbystaffwithexperienceofworkingwithsurveydata:

• oneinstitutionsupportedastudentresearcher,recruitedbytheSUinordertoscrutinisethecommentsfromtheNSS;

• anotherinstitutionincorporatedanalysisofNSSdata,andexaminationofthesurveyinstrument,ascourseworkonastatisticsmodule;

• severalinstitutionsdescribehelpingSUstouncoveryearlyprioritiesandobjectivesfromthedata;

• atoneinstitution,acourseleader‘commissioned’someresearchfromstudentrepresentativesworkingtogetherwithstaff,lookingatstudents’desireformorepracticalcontentinthecourse.

Beyondthis,afurtherlevelofco-operationcanexistwhereinstitutionalteamsandstudents’unionresearchersworktogetherinsingleteamstoexplorethedata.IncontrastwiththepartnershipsaroundrespondingthedatadescribedinSection2.1,thesekindsofcollaborationsondataexplorationseemrare.

The Students’ Union has recruited a researcher who has been scrutinising student data, including the NSS. He has been supported by the University analysis team in this, and is currently working with University staff in analysing the student comments, both at a top level separation into NSS categories and then doing a more thorough qualitative coding.

Case study: The National Student Survey at Queen’s University Belfast – building staff and student partnershipsTheresponseratetotheNationalStudentSurvey(NSS)atQueen’sUniversityBelfasthadplateauedat60%.Inordertobuildtheresponserateastudentpartnershipapproachwasestablished.ThepartnershipwasdrivenbythePVCEducationandStudents,theCentreforEducationalDevelopmentandtheStudents’Union.Anemphasiswasplacedupondevelopingstudent-ledfeedbackandutilisingstudentpeernetworkstoencourageuptakeofthesurvey.MirroringtheNSSStudentAmbassadorapproachStudentAmbassadorswererecruitedfromSchoolsandprogrammesacrosstheinstitution.StudentAmbassadorsweretaskedwithencouragingstudentstoparticipateinallformsoffeedbackandtoconsiderwaysinwhichtoclosethefeedbackloop.

Queen’sstudentscouldparticipateinfeedbackprocessesinanumberofwaysincludingmoduleevaluations,studentsurveysandclassrepresentativediscussions.ItwasnotedthatQueen’sstudentsdidnothaveaforuminwhichtoreflectuponanddiscusstheiruniversityexperienceswitheachotherandwithstaff.TheYourQueen’sExperienceConferencewasdevelopedtoaddressthisgap.ThisinitiativewasledbythePVCEducationandStudentsinpartnershipwiththeStudent’s

21

Union.StudentAmbassadorswerepairedupwithmembersofstafffromacrosstheinstitutiontofacilitatediscussionsattheConference.

TheConferencewasattendedby150studentsand50stafffromacrosstheinstitution.DiscussionscentredaroundthekeythemeswithintheNSSandreflecteduponNSSscoresandinstitutionalstrategiesandprogrammes.Akeyaimofthediscussionswastoensurestudentshadanaddedopportunitytoshapeinstitutionalpolicyandpractice.AsaresultofthesediscussionsandwiderstrategiesassociatedwiththeNSS:

• policyandpracticeinassessmentandfeedbackisbeingtransformedacrosstheinstitution;

• libraryprovisionandopeninghourshavebeenenhanced;

• studentconcernsandprioritiesareidentifiedearlierthroughinstitutionalsurveys,newapproachestotheevaluationofteaching,andnewcourserepresentativestructures;

• studentrepresentationhasbeenenhancedthroughincreasedparticipationinstudentelections,increasednumbersofstudentsvotinginelectionsandstrongercourserepresentativestructuresandenhancedtrainingprogrammes;

• studentengagementhasincreased;

• NSSresponserateshaveincreasedandscoresaremorerepresentativeofstudentopinion.

Maintainingstudentandstaffpartnershipsrequirescontinualattentionandpriority.Theinstitutionalcontextcanchangefromyeartoyearandrelationshipswithinthepartnershipcanchangefromyeartoyear.Thisisparticularlypertinentinthecontextofthechangingprioritiesandapproachesofelectedstudentrepresentatives.Inorderforthepartnershipstosucceeditisimportanttobeclearaboutrolesandresponsibilities.Itisnecessarytohaveclearstrategiesforcommunicationandaction.Itisimportanttotakethetimetoreflectonprogressandtoadjustandchangeapproacheswhennecessary.

ThepartnershipagreedthefollowingapproachtoaddressingandusingNSSscorestoenhancepolicyandpracticeatQueen’s:

• theprovisionofyear-on-yearcomparisonsofNSSresultstoSchoolsandacademic-supportunitsandtheStudents’Union;

• detailedcommentariespreparedforStudents’UnionandUniversitycommittees;

• institutionalprioritiesandstrategiesforimprovementdevelopedandidentifiedbyPVCEducationandStudents,CED,academicschoolsandstudents’;

• subjectspecificandthematicactionplansaredeveloped;

• NSSisintegraltomoduleandacademicprogrammereview;

• emphasisupon“closingtheloop”;

• courserepresentativestructuresprovideimpetusforbothaddressingandcommunicatingchange.

Forfurtherinformationcontact:nualatoman@hotmail.com

Nuala TomanQueen’s University Belfast

22

LessformalinvolvementofstudentsinexploringthemeaningandimplicationsofNSSresultsdoexist.SomeinstitutionsholdeventsandconferencesinspiredbytheNSSandotherstudentfeedback.

Theseeventsbringtogetherstaffandstudents,areoftenlooselystructuredaroundthefindingsofthedata,andcanhaveavarietyofexplicittopicssuchasrespondingtoNSSresults,preparingfornextyear’ssurvey,orspecificthemessuchasassessmentandfeedback.Thecommonfeatureisthattheyareusedtocreateaconversationbetweenstaffandstudentsaboutlearningandteachingissues.Notonlydotheyallowstafftohearfirst-handfromstudentsabouttheirexperiences,theyalsoallowstudentstoplayaroleinunpackingwhatthesurveyresultsmean.

Questions for reflection:

• DostudentshaveadirectroleintheexplorationandinterpretationoftheNSSresults,oristheirrolelimitedtothepromotionofthesurveyandattendingcommitteemeetingsasstudentrepresentatives?

• IssupportprovidedtotheSUtoenablethemtoundertaketheirownexplorationoftheNSSdata?

• Istherescopetorecruitstudents(includingpostgraduatestudents)aspaidresearchersanddataanalysts,ortoinvolvetheNSSinrelevantmodules,e.g.socialscienceresearchmethods?

23

2.3 Closing the loop

Key points

• FeedingbacktostudentsontheactionstakeninresponsetotheNSSisakeypartofengagingthemwiththesurvey.

• ‘Yousaid…wedid’providesaverydirectmethodoffeedback,buttheremaybealternativewordingsthatmorestronglyemphasisepartnershipanddialogueratherthanrespondingtoconsumerdemands.

• Ifthesurveydataarebeingusedappropriately,aspartofabroadersetofinformation,itmaybehardtolocatechangesmadespecificallyinresponsetotheNSS.Inthiscase,themessagethatisfedbacktostudentsmaybemoresubtleandcomplex.

SofarinthischapterwehaveseenhowstudentscanbeinvolvedbothindiscussingtheactionstakeninresponsetotheNSS(Section2.1)andinexploringandanalysingthedatathemselves(Section2.2).Thissectionlooksatanimportantstagelaterintheprocess:feedingbacktostudentsonthechangesthathavebeenmadeinresponsetotheNSS,or‘closingthefeedbackloop’.GivingstudentsaclearsensethattheirfeedbackhasbeenlistenedtoandacteduponisoneofthemostimportantwaystoengagestudentsintheNSS.Conversely,promotingthesurveytostudentsandthenfailingtoinformthemhowtheresultshavebeenusedcanleadtoresentmentanddisengagement.

We have a system where we give feedback to the students on the portal in terms of ‘last year you said in the NSS … and we have now done …’. I think this is the start of seeing the NSS as an enhancement tool rather than a league table instrument.

Institutionshaveemployedawiderangeofmethodsforgettingthemessagetostudentsaboutactionsthathavebeentaken:

• informationhasbeenincludedonwebsites,suchasmainuniversitypages,departmentalwebsitesandvirtuallearningenvironments;

• someinstitutionshaveinsertedactionstakenintostudentnewsletters,andmoduleandprogrammehandbooks;

• institutionshaveuseddedicatedformsofcommunication,suchasleafletsandmerchandise,designedspecificallytoconveyinformationaboutwhathasbeendoneinresponsetoNSSresults;

• publicdisplayshavebeenused,intheformofpostersandelectronicscreensoncampus;

• acommonactivityistoincludetheseformsoffeedbackaspartofthestandardpromotionofthesurveywheninvitationstorespondarebeingsentout.

Beyondtheserelativelystraightforwardformsofcommunication,someinstitutionshaveworkedwithstudentrepresentativesandtheSUtofeedbacktothestudentbody.Forinstance,atoneinstitutionanarticleforthestudentmagazineisco-writtenbytheSUandthelearningandteachingenhancementunit.

We work with the Students’ Union and talk to our student representatives to make them aware of the outcomes of the NSS and to enlist their support in raising awareness about the NSS and other surveys with their fellow students.

24

Othermethodsusedbyinstitutionsincludethefollowing:

• specificeventshavebeenruninordertodemonstrateanddiscusshowstudents’viewhavebeenrespondedto;

• oneinstitutionhaspresentedinformationabouthowfeedbackfromanumberofsources,includingsurveys,hasbeenrespondedtothrougha‘feedbackfortnight’;

• anotherinstitutionhasmadetheNSSactionplans–createdbydepartmentsinresponsetotheirNSSresults–publiclyavailable,sothatstudentscanseedirectlyandindetailwhatresponseshavebeenputinplace.

Thecontentofthesemessagesistypicallyfocusedontheactionsthathavebeentakeninresponsetotheresults,butinstitutionshavealsoprovidedinformationabouttheresultsthemselves,aswellasresponserates,togivestudentsasensethattheirparticipationhasbeenworthwhile.

Case study: Students’ opinions count at Cardiff Metropolitan UniversityOneareathathasbeenachallengeandoftenoverlookedbyinstitutionswithregardtotheNSSisthatofclosingthefeedbackloop.Thisisimportantforbothstaffandstudents.Thestudentsareoftenbombardedwithinformationaboutthesurveywhenthetimecomesforthemtocompletethesurvey,buthowmuchisactuallyfollowedthroughwiththechangesthataremadeinrelationtothecombinationofNSSscoresandopencommentsthatstudentsmakeregardingtheirsatisfactionwiththeiruniversityexperience.Althoughchangeswilloftenbemadeitisimportanttoletstudentsknowthatsomethingisbeingdonetoinrelationtotheirconcerns.Thisisessentialifstudentsaregoingtofeeltheyhaveagreaterownershipovertheirstudentexperience.Staffalsoneedtobemadeawareofhowatdifferentlevelsacrosstheinstitutionchangesaretakingplacesothatinformationcanbepassedontostudentsthroughthem.

TheinitialproblemisbeingabletoidentifyandattributechangesthathavebeenmadetotheNSSscores.Oncethiscanbeestablishedthenthenextstageistobeabletomakethestudentsawareofthechangesandinsomecasesjustsimplyexplainingwhycertainprocesseshavetobeadheredto.

AtCardiffMetropolitanwehaveworkedhardattryingtoidentifychanges.Thishasmeanttheintroductionofaprocesswhereassoonasresultsarereleasedtheyare1)analysed,2)discussed,3)actionplanscreated,4)plansprogressreportedonbothtostaffandstudents–inordertoclosethefeedbackloopstudentsandstaffhavetobeinvolvedatallstages.

WehavebuiltacloserelationshipwiththeStudents’UnionandtheyareanimportantpartofhelpingtopromoteallpartsoftheNSS.TheyareinvolvedingatheringstudentstogetherforthediscussionoftheresultsinSeptember/OctoberandtheSUpresidentattendsallmeetingsfromwhichinformationisusedfortheDeanofLearningandTeachingofeachindividualSchooltocreatetheiryearlyactionplan.

Theyarealsoinvolvedwiththelearningandteachingdevelopmentunitandmarketingdepartmentinputtingtogetherinformationonwhatchangeshavebeenmadeandwhereandhowtodisseminatethem.Thisoccursthrougha

25

numberofactivitiesincludingadvertsonTVscreensaroundallcampuses,regulararticlesinthestudentnewspaperonchangeandeffectsthatareoccurringandwheretheneedforthesechangeshascomefrom.TheSchoolRepresentatives’blogsarealsousedasavehicletoraiseawarenessanddiscussanyissuesandactionsthathavetakenplaceinresponsetotheNSSresultsanddiscussions.TheNSSisalsodiscussedatStudent-StaffCourseCommitteesandstaffarerequiredtocompleteasectionregardingtheirNSSscoresaspartoftheirannualevaluationofacademicprogrammereport.Examplesofactionsthathavetakenplaceandthatarediscussedatcommitteesandthroughthestudentrepsystemincludeimprovedallocationoffundingtothelibrary–notjusttheamountbutthewaythemoneywasspent–anincreaseine-journalsande-books,andtheinclusionofaonepagesynopsisregardinggeneralfeedbackforthewholeclasspriortofullindividualwritten/audiofeedbackinresponsetostudentsfeelingfeedbackwasnotpromptenoughtoassistintheirnextassignment.

Soithasmovedawayfromajusta‘Yousaid…wedid’processtoanannualcyclicalprocesswherestudentshopefullycanreallystarttofeelthattheiropinionscountandthattheycanbeinvolvedinchangeatallstagesfromstarttoend.

Nicola PooleCardiff Metropolitan University

Morecomplexthanthemethodisthetenorofthesemessages.Acommonapproachis‘Yousaid...wedid’aformofwordsthatsuccinctlycommunicatestheexistenceandnatureofaspecificactiontakeninresponsetostudentviews.However,someinstitutionshavemovedawayfromthattone,feelingthatitexpressesasenseofrespondingtoconsumerdemands.Institutionshavetriedanumberofalternativeapproaches:

• institutionshavetriedmessagesbasedontheideaoflistening,ratherthanconcretelyresponding:‘listeningtoyou’,or‘yousaid…welistened’;

• otherinstitutionshaveincludedspecificscoresandcomments,withsomegeneralinformationabouthowtheinstitutionisrespondingtothoseopinions.

Thesemovesawayfromthesimple‘Yousaid...wedid’areunsurprising,giventhetensionbetweenitsperceivedconsumeristtoneandtheincreasinglywidespreadsensethatstudentsarepartnersintheeducationalprocess,andthatgenuinedialogueismoreappropriatethanrespondingtodemands.Twoinstitutionshaveattemptedtobuildthatsenseofpartnershipintotheireffortstoclosethefeedbackloopbyundertakingactivitiesthroughouttheyear.Thisemphasisoncontinuousfeedbackanddialoguebetweenstaffandstudents,ratherthanaone-offmessagesentenmasse,isfelttopromoteandsupporttheideaofacommonlearningcommunity.

OneinstitutionusesoptionalquestionsintheNSStomonitorthatsenseofalearningcommunity;thereisagroupofthreequestionscalled‘feedbackfromstudents’,whichasksstudentsabouttheiropportunitiestoprovidefeedbackandhowthatfeedbackhasbeenvaluedandactedupon.Thisinstitutionfeelsthatthesequestionsprovideagoodmeasureoftheextenttowhichstudentsfeelengagedandinvolved.

The University communicates its use of the NSS data to the wider student body through a Facebook site called ‘Listening to you’, although this same site can be used to feed back to students on all changes made as a result of their input, i.e. the change may not have arisen exclusively from NSS feedback.

Asidefromcreatingandmaintainingatonethatfitswithhowtheinstitutionviewsitsrelationshipwiththestudentbody,thereareanumberofotherchallengesto

26

successfullyclosingthefeedbackloop.OneoftheseisduetothefactthattheNSSsurveysonlyfinal-yearstudents.Howdoyoufeedbacktostudentswhowillhavegraduatedandmostlikelyleftbythetimethatyouevenreceivetheresults?Therearemovesbeginningtoattempttodoso,aspartoftheprocessofconnectingwithalumni,buttheprimarycommunicationwilltakeplacewithsuccessivecohortsofstudents,thosewhowillthemselvessoonbeaskedtorespondtotheNSS.Theefficacyofthisislikelytodependonthesenseinwhichstudentsthinkofthemselvesasasinglecommunity.

The view from a students’ union – Cardiff Metropolitan University

CardiffMetropolitanUniversityStudents’UnionhasbeeninvolvedwiththeUniversityaroundtheNationalStudentSurveyinanumberofdifferentways,andtherelationshiphasgrownsincetheUniversity’sinvolvementatthepilotstage.Wearenowinvolvedwithbothexplainingthepurposeofthesurveyandencouragingstudentstocompletethesurvey(througharticlesinourstudentnewspaper,informationpassedonthroughourstudentrepresentativesystem,andjointtalkswiththeLearningandTeachingDevelopmentUnitonthepurposeofthesurveyto3rdYearstudents).Anotheraspectofthesurveyonwhichweplaceahighlevelofimportanceistheuseofresultstoenhancethestudentexperience.TheSUPresidentisinvolvedinthearrangementofmeetingsbetweenstudents,thePro-Vice-ChancellorandtheDeanofLearningandTeaching.Theyarealsopresentatthemeetingstohelpensurethevoiceofthestudentsisheardandtokeepabreastofanyissuesthatthestudentsmayraise.ThesediscussionsformpartoftheinformationthatDirectorsofLearningandTeachingusetocompiletheiractionplanstoaddressstudentissuesandimprovethelearningexperience.ThePresidentandtheVice-PresidentalsositonanumberofBoardsincludingtheBoardofGovernorsandLearningandTeachingBoard,andareinvolvedinNSSdiscussionsatalllevels.

Ruth FosterPresidentCardiff Met Students’ Union

Asecondchallengerelatestoanissuethatwillbethefocusofalatersectionofthisreport(Section4.2),theneedtousetheNSSasoneofseveralsourcesofdata.TheNSSdoeshavemanyvirtuesasasourceofinformationaboutstudents’experiences,butitcanneverprovideacompleteorprecisepictureofwhatisgoingon.Itfunctionsbestasastartingpointforfurtherinvestigation,andaspartofatriangulationprocesswithothersourcesofinformationsuchasinternalsurveys,informalconversationsanddialoguewiththestudents’union.ThiscreatesatensionwiththeneedtoprovideinformationtostudentsabouthowNSSresultshavebeenactedupon.SeveralinstitutionsacknowledgedthedifficultyofreportingactionstakenindirectresponsetotheNSS,whenchangesarelikelytohavearisenfromconsiderationofarangeofdata.

Concrete changes that can be directly attributed to the NSS are more difficult to substantiate. Changing practice is part of an ongoing process of dialogue with various stakeholders within the institution, including students, and the NSS is only one contributory element that informs the process of dialogue.

Questions for reflection:

• ArestudentsawareofthevaluethatisplacedontheNSSresults,andtheimpactthatthoseresultshaveondecisionswithintheinstitution?

• AretheoptionalquestionsintheNSS,thataskstudentsabouthowthefeedbackthattheyprovideisrespondedto(QuestionB.6),usedwithintheinstitution?Ifnot,doyouhaveotherwaysofgatheringthatinformation?

27

• IfyouareunabletociteactionstakenindirectresponsetoNSSscores(becausedecision-makingismorecomplex)howelsecanyoumakecleartostudentsthatthesurveyresultsaretakenseriously?

• Doesthetoneandmethodbywhichthemessagesarefedbacktostudentsembodytheinstitution’svisionofthestaff-student,andinstitution-student,relationship?

28

3. Institutional structures

3.1 Staff attitudes

Key points

• ThecommonperceptionthattheNSSisa‘sticktobeatuswith’canbemitigatedbycapturingandcommunicatinggoodpracticewhereresultsarebetterthanexpected,andnotjustfocusingontheareaswithdisappointingsurveyscores.

• JustasinvolvingstudentsintheNSScanbebeneficial,engagingstaffatalllevelsintheprocessofpromoting,exploringandrespondingtotheNSSisakeypartofusingthesurveyforenhancement.

• TheNSScanoftenbeapowerfulbutunspokenpresence;therearebenefitsinbringingitoutintotheopensothatitsstrengths,limitationsandimpactcanbehonestlydiscussed.

TheuseoftheNSStocreatepositivechangestolearningandteachingdependsonthewillingnessofstaffatalllevelstobeinvolvedintheprocessofdiscussingandactingonstudentfeedback.TheNSScansufferfrombeingviewedasamanagerialandbureaucraticexerciseinbox-ticking,andsomeinstitutionsreportthatwhileasmallnumberofstaffmayenthusiasticallyembracetheuseofdataaboutstudents’experiences,therehasoftenbeenlittleinterestamongacademicsingeneral.However,thereisalsoasensethatthisischanging,andthattheincreasedfocusonstudentchoiceandinformationaboutHE(andtheKISinparticular),isincreasingthevisibilityoftheNSSthroughoutinstitutions.

There is a mantra, ‘we might not like it, but it’s here, so deal with it’. Ignoring or dismissing the NSS, perhaps a common response of teaching staff in the earlier days, is no longer seen as an option.

AcommonviewisthattheNSSisclosetotheheartofstudents,whilebeingdistantfromthelivesofacademics.Oneinstitutioncommentedthattherealityisthereverse:moststudentscompletethesurveysimplyduetothelevelofremindersthattheyreceive,andinfactdonotinvesttheirresponsewithagreatdealofemotionorattention.Academicstaff,ontheotherhand,especiallythoseinmanagementpositions,canfindthattheNSSisacentral,highlyvisible,andoftenemotiveelementofuniversityanddepartmentalprocessesduringspecificpartsoftheyear.

ThislevelofvisibilitythattheNSSpossessesamongstaffdoesnotnecessarilyequatetohighlevelsofpositivityorenthusiasm,andoneofthemostwidelyexperiencedobstaclestostaffinvolvement,reportedbymostinstitutions,isthesensethattheNSSisusedasasticktobeatdepartmentsandfaculties,ratherthanasapositiveencouragementtoimprove.Thisdeficitmodelmaybeprojectedthroughthekindsofactivitiesthatcanbestandardpartsoftheplanningandreportingcyclediscussedlaterinthisreport,inSection3.2:

29

• focusingonpoor-performingareas;

• requiringdepartmentstowriteactionplansaddressingweakerscores;

• emphasisingthenegativeimpactoflowscoresonissuessuchasrecruitmentandretention.

Theseactivitiesareundoubtedlynaturalconsequencesofthepressurethatseniormanagersthemselvescomeunder,whichisoftentotakeanoutward-facingperspectivethatfocusesonleaguetablesandothermetrics.Notonlydoesthisdemoralisestaffandcreateanaturalsenseofresentment,someinstitutionsreportthatitalsoleadstoadefensiveattitude;NSSdataarenotseenasusefuldatathatcanbeusedinconjunctionwithotherinformationinordertoinformenhancementactivities,butassimplemeasurestobedefended,explainedaway,underminedordismissed.Oneinstitutioncitedthedamagingsituationwherestaffareaskedtorespondtoverysmall–infactstatisticallyinsignificant–differencesinresults,becauseofaneffectontheleague-tableposition.Anotherinstitutionreportedpressurebeingplacedondepartmentsbecauseofadropinrankings,whenthatwaspurelyduetotheimprovementofothers,ratherthanadecreaseintheirownscores.

The way the NSS is managed nationally and by senior managers within universities creates a sense of imposition and inconvenience that is difficult to get past.

Asinstitutionshaverecognisedthisperceptionofadeficitmodel,manyhavetriedtoaddresstheissue:

• sometryandpromoteafocusonareasofgoodpracticeaswellasareasofconcern,toconveythesensethatdepartmentscanlearnfromeachotherandsharethingsthataregoingwell,toaddressthingsthataren’t;

• oneinstitutionhasexplicitlyfocusedontheareaswiththemostpositivescores,andencouragedstaffinthoseareastocreatecasestudiesthatcanbesharedthroughouttheinstitution;

• anotherinstitutionrunsworkshopsthatbringtogetherthehigh-andlow-performingdepartmentsinordertosharegoodpractice;

• anotherapproachhasbeentomakewideruseofthetextcommentsyieldedbythesurvey.Thefactthatsuchdatadonotlendthemselvestosimplebetter/worsejudgementshashelpedtosoftenthesensethattheNSSlabelsdepartmentsasweak.Thecommentsmadebystudentscanalsohaveamoreformativefeel,astheyaremorepersonalandengaging,andcontainstudents’thoughtsabouthowthingscouldbeimproved.Theemotivenatureofthecommentsmadebystudents,althoughtheyneedtobeinterpretedandusedwithcaution,canopenupdiscussioninawaythatthenumericalscoresoftencannot.TheuseofqualitativedataisdiscussedfurtherinSection4.4.

ThatprocessofengagingstaffindiscussionsaroundtheinterpretationanduseofNSSdatahasbeenfoundbysomeinstitutionstobeakeypartoftheeffectiveuseoftheNSSasanenhancementtool.JustaswehaveseeninChapter2howengagingstudentswiththeNSSiskeytoexploitingitspotential,itseemsthatstaffengagementcanmakethedifferencebetweentheNSSsupportingasimplisticmechanismforqualityassuranceandsupportingasophisticated,informedandpositiveprocessofqualityenhancement.

30

Research study: Academics’ views on the NSS

Adam Child reflects on his research into academics’ views about the NSS

In2010-11IundertookastudydesignedtoexploretheviewsofacademicstafftowardstheNSS,particularlyasapotentialtoolfortheenhancementoflearningandteaching.Over300stafffrom12pre-1992universitiescompletedaquestionnairethatposedaseriesofquestionsabouthowtheNSSwasusedwithintheirinstitution,andaskedrespondentstoratetheextentoftheNSS’simpact.ThiswasparticularlyinterestinggiventhediscussionstakingplaceatanationallevelonthefutureoftheNSSandthepossibleintroductionofasimilarsurveyforpostgraduatestudents.Iwasespeciallycurioustodetermineiftheconclusionsofthe2010evaluationoftheNSSwerecorroboratedbyabroaderstudy.

ItwasclearfromtheresponsesIreceivedthattheNSSisperceivedasalargelytop-downconcern,withthemajorityofrespondentsstatingthattheimpulsetorespondtotheNSScomesfromthemanagementwithintheirinstitution(looselydefinedasthosewithseniorityoutsidetheirowndepartment),andthattheNSSisofmoreinteresttotheseinstitutionalmanagersthantoindividualteachers.Thisislikelytoberelatedtothedesireofaninstitutiontoappearfurtherupuniversityleaguetables,mostofwhichplaceaheavyemphasisontherawscoresofthesurvey.Itwasapparentfromthecommentssubmittedthatleaguetableswereattheforefrontofpeople’sthinking.WhatisalsointerestingisthewayinwhichthisemphasisonmediapresentationisactuallypreventingmoremeaningfulengagementwiththeNSS(andotherinternalsurveys)fromtakingplace.AkeyfindingofthestudyisthattheroleoftheNSSasapublicperformanceindicatorcanhamperitsmorerecentlyemphasisedpurposeasatoolforenhancement.

Ialsofoundthatlessthan10%ofrespondentsagreedthattheNSSwastheirpreferredmethodofgatheringstudentfeedbackandover70%feltthattherewereothermoreappropriatetoolsforobtainingstudentviews.ThissuggeststhatakeywayofusingtheNSSistoincorporateadiscussionofthesemetricsintoawiderdiscussionaboutprovision.AsthedataavailablefromotherformsoffeedbackarelikelytobedepartmentallyorprogrammefocusedthisinturnmeansthatthediscussionsaroundtheNSShavetobeatthislevel.AtLancaster,myowninstitution,thisisfacilitatedthroughannualteachingreviews,wheremanyotherformsofdataareprovidedtodepartmentsalongwiththeirNSSscores.DepartmentsareabletotakewhateveractiontheydeemnecessarytorespondtotheNSSwithintheirwidercontexts.

ThisresearchraisessomeissuesabouthowtheNSSisperceivedandhowthisimpactsonitsuse.ThereisnowempiricalevidencedemonstratingthatnegativestaffperceptionstowardstheNSSexistandcanadverselyaffectengagementwiththesurvey.Italsopointstowardsstrategiesinstitutionscanusetoincreaseengagement:informandpersuadecolleaguesofthebenefitsofusingtheNSSalongsideotherinformationandincollaborationwithstudents.AkeytothisappearstobebreakingthelinkbetweentheNSS’smultiplefacets,separatingitsuseasasimplisticindicatorofperformancefromitspotentialasarigorousquantitativesurveyofstudentperceptions.Achievingthissegregationmaynotbestraightforward,butifitisachievedwewillbeabletostartdiscussionsonamuchmorepositivefooting.

Adam ChildLancaster University

ThisresearchwaspresentedattheSurveysforEnhancementconferencein2011,pleaseseehttp://www.heacademy.ac.uk/resources/detail/evidencenet/Rage_against_the_machine[accessed15October2012].Afullaccountoftheresearchcanbefoundathttp://etheses.whiterose.ac.uk/2424/[accessed15October2012].

31

OneinstitutioncommentedthatitisimportantthateffortstoinvolvestudentsintheuseoftheNSSdonotleadto‘front-line’staffbeingsidelined.‘Staffengagement’asanoverlookedcounterpartofstudentengagementisanincreasinglycommonthemeofdiscussionsaroundqualityenhancement.AninstitutionsuggestedthataNationalAcademicStaffSurvey,exploringacademics’perceptionsbothoftheirstudents’experiences,andofthepressuresoftheirownworkinglives,couldbethenextsteptoconsider.

Onepartoftheprocessofengagingstaffishelpingtocreatedialoguebetweenstaffandstudents.Aninstitutionhasdonethisbyinvitingstafftocompletethesurveythemselves,reflectingonhowtheywouldexpecttheirstudentstorespondtothestatements.Thestaffandthestudentshavethendiscussedtheirresponses,asawayofexploringinterpretations,expectationsandultimatelythestudents’experiencesoftheircourses.

There has been a shift in attitude about the NSS over the past couple of years. Academics now see this as a valuable source of information on which to base enhancement activity. In the early days of the NSS it was viewed with more suspicion.

AcommonchallengecitedbyinstitutionsisthelackofopendiscussionabouttheNSSandthedataityields.Thosedatamaybewidelyreportedandincludedinthedecision-makingprocesseswithoutbeingexplicitlyaddressedinconversations.Toovercomethis,severalinstitutionshaveattemptedtocreateanenvironmentwhere,whilethepresenceoftheNSSasafactoflifeisacknowledged,therecanbeopendiscussionofthemeritsandlimitsofthesurvey.Theyfeelthatifstaffcanraiseconcernsaboutaspectsofthevalidityofthedata,andcanreceivehonestanswersbasedonevidence,itmaybeeasiertopromotethemessagethattheNSSissimplyagoodsourceofdatatobeusedalongsideothers,ratherthananaccuratesummativeassessmentofthequalityofacourse.

We try to remove the previously negative stigma attached to poor management of the data and promote it as much more of a data set to be engaged with at many levels, to better understand deficiencies or areas where the sharing of practice could benefit the student experience.

Alongsidethismoreopenapproach,someinstitutionshavefound,unsurprisingly,thatstaffaremorewillingtoengagewiththeNSSifarigorousevidence-informedapproachisadopted.Acriticalbutconstructiveattitudeisnotonlythemostappropriatewayofusingdata,itisalsobehaviourthatacademicsexpectwithintheirdisciplines.Thismeansthatnotonlyshoulddataanalysisbeperformedaccuratelyandappropriately,butthepresentationandfurtheruseofthedatashouldbefullyinformedbytheresearchthatdoesexistonthestrengthsandlimitationsoftheinformationyieldedbytheNSS.Oneroutethatinstitutionshavetakenistoacknowledgetheimportanceofusingmultiplesourcesofdata,nottheNSSalone,whenprovidinginformationtostaff.ThisissueisthefocusofSection4.2.

ManyinstitutionshavefoundthatoneofthekeywaysofencouragingdepartmentstouseNSSdataistoensurenotonlythattheanalysisis–andisseentobe–robust,butthatdataarebrokendowntotheappropriatelevel.Effortexpendedonprovidingfine-graineddetail,atthelevelofdepartmentsandcourses,hasprovedworthwhileingeneratingstaffengagement:

• oneinstitutionhasfoundsuccessindevelopingaformatthatallowscourseteamstoquicklyreviewtheirdatayearonyear;

• otherinstitutionshavefoundthathavingdedicatedandpublicisedpointsofcontactforNSSquerieshassmoothedtheprocessofprovidingaccurateandtargeteddata.

There was a considerable level of distrust of the results when they were not available at a fine grain of detail. The breakdowns to small department areas and courses has done much to encourage engagement.

32

Moregenerally,afewinstitutionsreportthatbygivingstaffsomeownershipoverthewholeNSScycle–thepromotion,analysisandresponse–staffengagementhasincreased.Oneinstitutionexplicitlyreducedbureaucracyinordertoallowacademicsthespacetothinkcriticallyaboutstudentfeedback,andtocomeupwithinnovativeresponses.Giventheimportanceofstaffinvestmentforthesuccessofanyattempttoimprovestudentlearningthroughrespondingtofeedback,institutionsuniversallyreportthateffortstoinformstaffaboutthepotentialoftheNSS(aswellasitslimitations)areworthwhile.

Questions for reflection:

• Doestheinstitutionrespondtopoor-scoringareasintheNSSinasupportiveway?Areareasthatconsistentlyperformwellinthesurveyhighlighted?

• HowmuchfreedomdoacademicshaveabouthowtheyrespondtoNSSresults?Aretheyconstrainedbyanoverlybureaucraticprocedurethatpreventsthemfromengagingandinnovating?

• IstheNSSopenlydiscussedanddebated?Istherelevantresearchliteraturereadandapplied?

33

3.2 Integration into decision-making

Key points

• IntegratingtheNSSintoplanningandreviewcyclescanpromoteamorereflectivequalityenhancementapproach.

• WheninstitutionsusetheNSStocreatetargetsandKPIs,itisimportanttobearinmindthelimitationsofthedata,andtheneedtotriangulateNSSscoreswithothersourcesofinformation.

InthelastsectionwesawhowtheNSSwasbecomingmorevisibletoacademicstaff,andhowinstitutionsweretryingtoengagestaffwiththesurveyasatoolforenhancement.Oneelementofthat,alsopromptedbytheincreasingimpactoftheNSSonstudentchoice,isitsintegrationintotheinstitutionalstructuresbothforreviewingpreviousactivityandplanningforthefuture.Ithasbecomeakeysourceofinformationwhenmakingdecisions,andakeysourceofbothevaluationinformationandperformancetargets.

Oneprominentchangehasbeenthedevelopmentofanannualplanningandreviewingcyclerelatedtothesurveyitself.Thistypicallyincludesmonitoringofresponserates,performanceagainstcompetitors,andyear-on-yeartrends.Atsomeinstitutionsitalsoinvolvesvisitsbythepro-vice-chancellororevenvice-chancellortodepartmentsorfaculties,todiscussNSSresponseratesandresults.

The NSS is promoted as an enhancement tool through its integration with the annual planning and review cycle.

InadditiontothisdiscreteNSScycle,resultsareusuallyintegratedintothewiderreviewprocesses.InstitutionshaveincorporateduseofNSSresultsintoawiderangeofprocesses,includingmodulereview,programmereview,andreviewsofenhancementactivities.Thesereviewsdon’tonlyinvolveNSSscores,butsuchscoresnowoftenformakeypartofthedatasetthatteamsarerequiredtoconsider.Severalinstitutionsarefindingsuccessinprovidingexamplequotesfromstudents’commentsforteamstoreviewalongsidethenumericalscores.Whiletheuseofstudentfeedbackinthiscontexthasperhapsbeenlargelyoptionalorimplicitformanyyears,whatdoesappeartohavechangedisthatinmanyinstitutions,theexplicitconsiderationofNSSresultshasbecomearequirement.

The NSS has changed institutional practice in that there is now explicit discussion of NSS results as part of the University’s rolling review processes at school, faculty and central committee level.

NotonlydoestheNSSnowplayaroleinmoduleandprogrammereviews,itisalsoplayingapowerfulpartindeterminingstrategicdecisionsatinstitutionallevel.SeveralinstitutionsreportthattheirkeyperformanceindicatorsexplicitlyrelatetotheNSS,e.g.anaimtoimprovesatisfactionlevelsyearonyear,ortobewithinaparticulargroupofinstitutions(suchasthetopquartile)nationally.Thismustbedonecautiously,giventhereflectionsinSection4.2thattheNSSshouldnotbeusedinisolationfromothersourcesofdata.SomeinstitutionsalsousetheNSStoidentifycross-institutionalthemesforconsistentaction,suchasassessmentandfeedback.TheinclusionofNSSresultswithintheseprocessesisseenbysomeasanincreasingandwelcomefocusontheenhancementofquality,alongsidemorerigidqualityassuranceprocesses.Someinstitutionsreportthattheconsiderationofstudentviewscanfostermorereflectiveenhancement-ledapproaches.

Through its use of the NSS, the University has been able to address both the KPI/business objective aspect of the NSS, and the more reflective quality enhancement issues. While these are undeniably linked there are important distinctions between the two.

34

OneofthemostuniversalandprominenteffectsoftheNSSwithinplanningandreviewprocessesisthecreationofNSS-specificactionplans,andtheincreasingrelevanceofNSSresultstogenerallearningandteachingactionplansatalllevelswithininstitutions.

ThedecisionwhetherornottorequirededicatedactionplansforNSSresultsisnotnecessarilystraightforward.Atleastoneinstitutionhasmadeadecisionnottodoso,inordertomakesurethatstafffocusonwiderstrategicissuesratherthanthenarrowissueofspecificsurveyscores.Attheotherextreme,anotherinstitutionhasfeltitbeneficialtoharnessthevisibilityandconsequentleverageoftheNSS,bymakingthesurveycentraltothewiderenhancementactionplanningprocess,eventotheextentofcallingthosewideractionplans‘NSSactionplans’.ThisissueofwhetherthereareseparatestructuresfortheNSS,orwhetheritisembeddedintothestandardprocesses,isalsoraisedbythefactthatsomeinstitutionshavecreated‘studentvoice’posts,orevenentireunits,dedicatedsolelytotheuseofstudentfeedback.Oneinstitutionhassetupa‘studentvoice’programme,supportedbydedicatedstudentrepresentatives,whoareinturnsupportedbythestudents’unionandtheinstitutionalQualityOffice.

Thespecificmechanismsofactionplanningvarymarkedly,butherearesomeexamples:

• severalinstitutionsdrawonarangeofnumericaldata,includingNSSresultsbutalsoresultsfrominternalsurveys;

• oneinstitutionhasinstigatedNSS‘actiongroups’,toallowlargerteamsthannormaltoconsiderhowtorespondtostudentfeedback;

• insomeinstitutions,therequirementtoproduceactionplansistargetedinspecificways,commonlyondepartmentswithdisappointingscores;

• oneinstitutionencouragesdepartmentstothinkaboutthemessagetheywanttocommunicatetostudents,ratherthantreatingactionplanningasapurelystaff-focusedprocess;

• asdescribedinChapter2,someinstitutionsexplicitlyinvolvestudentrepresentativesintheactionplanningprocess,forexamplebyencouragingthemtocommentondraftplans,orbypublishingalldepartmentalactionplansontheuniversitywebsite.

AstheNSShasincreasedinprominence,itsrolewithindecision-makingprocesseshasbecomebothmoreformalisedandmorepronounced.Inmostinstitutions,departmentsandfacultiesarenowrequiredtorespondtotheirNSSresults,especiallywherethoseresultsaredisappointing.TheNSSseemstohavehelpedtoincreasetheprevalenceofannualcyclesforthemonitoringandimprovementofeducationalquality.

Questions for reflection:

• IstheNSSembeddedinyourplanningandreviewsystemsinawaythatpromptscriticalreflectionandinnovationbycourseteams?

• AreacademicsencouragedtousestudentcommentsfromtheNSSalongsidethenumericalscoreswhenreviewingmodulesandcourses?

• DoinstitutionalNSStargetssufficientlytakeintoaccountthelimitationsofthedata?

35

3.3 Impetus for change

Key points

• TheNSShashelpedtocreateanewfocusonthestudentlearningexperience.

• ThesimplisticuseoftheNSSinthemediaisadouble-edgedsword.Whileitcausesresentmentandsometimesanarrowfocusonraisingscores,italsocreatesaleverforchange.

• IndividualsworkingtoenhancelearningandteachingcanuseNSSresultstopersuadecolleaguesoftheneedforchange.

ItisnowacommonplacethatNSSresultsareresponsibleforconcretechangeswithininstitutions.Therearenumerousexamplesofinstitution-wideinitiatives,designedandimplementedinresponsetoNSSresultsviewedaspoororunsatisfactory.WehaveseeninSection3.2howthesurveyisincreasinglybuiltintoplanningandreviewsystems.

WhydoinstitutionsfeelcompelledtoinvestsucheffortinreactingtoNSSscores?ThealmostuniversalmessagefrominstitutionsisthattheadventoftheNSS,andthehugenumberofresponsesitreceives,hashelpedtocreateamorepowerfulfocusontheimprovementofthestudentlearningexperience.Thisaddedfocusisalsoapparentatthenationallevel,withgovernments,sectoragenciesandtheNationalUnionofStudentspayingnewattentiontothequalityofeducationreceivedbystudents.Althoughthereisalotofinstitutionalvariation,acommonperceptionisthatlearningandteachinghavemovedhigheruptheagenda.

One of the benefits of the NSS is a national and institutional focus upon learning and teaching, student experience and student satisfaction.

InstitutionsgenerallyfeelthattheimpactoftheNSSisduetotheroleoftheresultsinthepublicperceptionofinstitutions.This,inturn,isfelttobeduetotheuseofNSSrankingsinthemedia,andtothecontributionoftheNSStothevariousleaguetablesintendedtoinfluenceprospectivestudents.FromSeptember2012,theKeyInformationSetsareexpectedtoincreasethisaffect.NotonlydothesemechanismscreateageneralmotivationtomonitorandimproveNSSscores,theyarealsoaddedpressuresonthoseinstitutionswithdisappointingrankingsorleaguetablepositions.

AswesawinSection3.1,theimpactofleaguetablesoninstitutionalprioritiescanoftenbeasourceofdiscomfortandnegativity.Staffareresentfulofhavingtorespondtosimplisticmetrics,andtotheideathatthemarketing‘tail’iswaggingtheresearchandteaching‘dog’.OneaspectofthisconcernisthatNSSresultsareusedtotheexclusionofotherinformationaboutstudents’perceptions,andcandefineandlimitinstitutionalenhancementactivitiesinwayswhicharenotalwayseducationallyappropriate.AsweshallseeinSection4.2,NSSresultsonlyprovideaninitialoverviewofhowstudentsareperceivingtheircourses,andtheycannotbearthiskindofweight.

Thisreputation-ledconcernforNSSresultscan,however,haveapositiveimpactonqualityenhancement,astheattentionthatitreceiveswithinaninstitutionmayprovideasourceofleverageforchange.SeveralinstitutionshavefoundthatNSSresultscanbeusedtostart(oraccelerate)enhancementactivitiesthatotherwisewouldnotreceiveattention.Seniorstaff,includingpro-vice-chancellorsandvice-chancellors,mayoftenbemotivatedbyNSSresultswhenotherinformationhasfailedtohavemucheffect.Whilesomeinstitutionsbelievethatthispositiveoutcomeforqualityenhancementhasbeenanaccidentalby-productoftheNSSratherthananintendedresult,thereisabroadconsensusthattheNSShasbroughtgenuinebenefitsforlearningandteaching.

36

Theexternalprofilethataninstitution’sNSSresultshave,throughleaguetablesandothermechanisms,thusprovestobeadouble-edgedsword.Whiletheinappropriateandsimplisticuseofthedatahascreatedrealresentmentamongstaff,aswellascausingeffortandresourcestobemisdirected,thereisalsoanacknowledgementthatwithouttheexistenceoftheNSS,therearelikelytohavebeenfewerconversationsaboutlearningandteachingwithinUKinstitutions.

The fact that the data is published, and informs league tables, helps those of us who advocate quality enhancement to persuade colleagues of the value of bringing about change.

Case study: Using the NSS at the University of Glamorgan - slightly easier than climbing EverestTheUniversityofGlamorganhasaround16,500undergraduates;bringingitsNSS‘sample’toaround3,500.Itsoundsfairlystraightforward,butthe‘shape’ofGlamorganturnsawalkintheparkintoanascentofEverest!Undergraduatesarespreadacross14locations–includingtheRoyalWelshCollegeofMusicandDrama,aseparateHEI(butnotforNSSpurposes).SoascendingEverestiseasyincomparison.NSShighresponseratesandeffectiveactionplanningisnot!

Duringthesurvey,amultitudeofactionsaretakentoraiseresponseratesinthiscomplexinstitution–onesuchistherovinggiantgingergoatcalledGandalfwhowanderswitha‘minder’andWi-FilaptopfortheuseofNSSstudentswhodawdlelongenoughtostareatthismascotoftheStudents’Union!

UsingtheresultsisalittleeasierthanclimbingEverest.Forthreeyearssystematisedactionplanninghaslookedbackwardandforward,Janusfashion.Whatworkedlastyear?What’llworknow?Faculties,Students’Union,theCollegeandLearningSupportteam(IT/library)areprovidedwithactionplanning‘kits’ofstatisticalreports,which,takenasawhole,shouldpinpointmostissues.TomakelifeeasierforcolleaguesatthattaxingpointintheAutumnterm,somesatisfactionreportsareevenproduced‘numberless’butusingatrafficlight‘picture’ofwherethesubjectfitsincomparisontoitssectoraverage.AllactionplanningiscompletedbyearlyNovember,withtheLearningSupportTeam’splancomingalongslightlylaterasitssupportingactionsunderpinacademicplans.Allplansinvolvediscussionwithstudentreps.

ThefullestinvolvementofstudentsintheuseoftheNSSischallenging.Askingstudentswhatmakesthemsatisfiedgeneratesinevitableresponsesaboutacresofcarparksandswimmingpools.Aswequestion,sowesetexpectationsrunningandthisismanaged(ifpossible)throughcontinualdialogueabouttheNSS.Lookingbackto2005,thesurveyhasnudgedGlamorganoutofits‘comfy’positionwithstudentrepsintofuller,moremeaningfulengagementateverylevel.UsingtheresultsasavehiclefordialoguehasledtoaclearerunderstandingbetweentheUniversityanditsstudents.Thedata(withothermetrics)arenowthecornerstoneofannualmonitoring,curriculumplanning,qualityreviewsandspaceutilisation(yes!).Businessprocesses,too,havechanged,suchasthepublicationofgradesandchangestoenrolment.Glamorganwillgofurther:anideapickedupattheHEA’sSurveysforEnhancementconferencehaspromptedtheinclusionofactionsoriginatingwiththestudentsthemselvesandtheUniversityiskeentoseehowthispansout.

37

Real,practicaluseofthesedatamakesadifference,anditcertainlydoestostudents!Althoughsustainedimprovement,bydefinition,can’tbequick.SotheUniversitywillcontinueusingNSSresultsasoneofitstools(it’sthedeluxemonkeywrenchofatool:strongest,mostreliable,effective,andalwaysthere).Therearesignificantothers:valued,listened-tostudentreps;responsivestaffandsystems;andmoderatelysimplethingssuchasastabletimetable.Nothingthat’sworthwhilewasevereasy,assomeoneoncesaid.

Denize McIntyreUniversity of Glamorgan

HowarepeopleusingthisabilityoftheNSStoinitiateandsupportchange?Someinstitutionshavetriedtomovebeyondthekindsofdamage-limitationexercisesrequiredbyinstitutionalmarketing.Instead,thereisarecognitionthatwhatisoftenneededistogenuinelychangethediscourse,todevelopaculturewherestudents’needs,learningandteachingareopenlydiscussed.TheNSSprovidesafrank,ifunfinished,pictureoftheexperiencesofstudents,andthishasbeenusedbysomeinstitutionstohelpfosteramorestudent-centredfocus.AswesawinChapter2,theNSShasbeenusedtocreatebetterpartnershipsbetweentheinstitutionandstudents,andtheirrepresentatives.Inshort,althoughtheNSSwasdesignedandimplementedlargelyasasummativemeasureofstudentperceptions,therearemanyinstitutionsattemptingtouseitinamoreformativeanddevelopmentalway,asafocusforfurtherinvestigationandaguidetoenhancementefforts.

It’s not just about marketing but about changing the discourse.

Manyoftheseinitiativesarefocusedonissuesaroundassessmentandfeedback.Studentsarealmostuniversallylesspositiveaboutthoseareas,andinstitutionsoftenreactbydevelopingnewsystemsforcreating,providingandmonitoringfeedback.Large-scalereactionstotheNSSarecommon,butoneinstitutionhascommentedthatnotallchangesare,orneedtobe,soall-encompassing,andsomeinstitutionshavefocusedinsteadonsmaller,morelocalconcerns.Alessonlearntbyoneinstitutionisthatinterventionscanoftentaketime.AlthoughsomeresponsestoNSSscoresmayyieldimmediatebenefits,moreoftenanimprovementinthenextyear’sresultsisunlikelytooccur.Importantchangescanbemade,butpatiencewilloftenberequired.

A lot of changes have been made as a result of the NSS data. It is very important to say that not all of the changes need to be large scale projects.

TheNSSisnottheonlyfactorthathascontributedtoagreaterfocuswithintheUKHEsectoronthestudentlearningexperience.Otherdevelopmentsinclude:

• changestothenationalfundingsystems;

• arequirementbytheQualityAssuranceAgencyforgreaterfocusonpublicinformation;

• theincreasingsensethatstudentsarepartnersintheeducationalprocess;

• anincreaseinthemarket-likepropertiesofthesector;

Thesechangeshaveallcombinedtofocuspeople’smindsonthewaysthatstudentsareexperiencingtheirlearning.Whiletherearerisks,complicationsandnegativeconsequencesofthesechanges,thereareexamplesofinstitutionsusingtheincreasedfocusonthestudentexperience,partlyduetotheNSSbutalsototheseotherdevelopments,tomakegenuineimprovementstohowtheirstudentslearnanddevelop.

38

Questions for reflection:

• HastheNSScreatedagreaterinstitutionalfocusontheenhancementoflearningandteaching?Oristhemainimpactonmarketing,recruitmentandqualityassurance?

• HasthenetimpactoftheNSSonstudentlearningbeenpositiveornegative?

• HastheNSScontributedtoa‘studentasconsumer’approachwithintheinstitution,orhasithelpedtosupportagenuinedialogueandengagementwithstudents?

39

4. Analysis and exploration 4.1 Limitations and challenges

Key points

• ItiscrucialthatthereisawarenessandhonestyaboutthelimitationsoftheNSSasatoolforenhancement.

• TheNSSonlyprovidesapartialpictureofstudents’experiences,andneedstobesupplementedinordertobeuseful.

• TheNSSgivesprominencetostudentsatisfaction,butitisimportantthatinstitutionsgatherdataonotheraspectsoflearning,inordertohavearoundedviewofeducationalquality.

Aswehavealreadyseeninthisreport,theNSShasthepotentialtomakeausefulcontributiontotheenhancementoflearningandteaching.However,aswithanyresearchmethod,therearealsolimitationsthatneedtobeacknowledgedanddiscussedsothattheyaremitigatedwherepossible.Thissectionwillnotaddressthesurveytoolinitself,suchasthevalidityorreliabilityofthesurvey,butwillfocusontheissuerelevanttothisreport;theobstaclesthatarisewhenusingitforenhancement9.

I think there is always value in collection of survey data on this kind of scale – provided we remain conscious of the limitations of such information and utilise it wisely.

Wehavejustseen,inSection3.3,howtheuseoftheNSSinthemediaisadouble-edgedsword.Whileitdoescreateanimpetusforchange,itcanleadtoresentmentanddisengagementamongstaff.Section3.1describedhowattentiontoleaguetablesandmediarankingscancreatetheperceptionofadeficitmodel,whichignoressuccessandgoodpracticeandfocusesonfailure.Thisisaclearobstacletotheuseofthesurveyaspartofareflectiveandpartnership-basedapproachtoqualityenhancement.ThosesectionsalsocontainideasforminimisingtheeffectofthatkindofsimplisticuseoftheNSSinthemedia.

AnotherobstacleistheideathattheNSScanbeusedinisolationasanaccuratemeasureofthequalityofstudents’educationalexperiences.ThisviewoftheNSSispartiallycausedbythewayitisemployedinthemediatomakesimplecomparisons,butthesurveycannotbeartheweightofbeingusedasastand-alonemeasureofquality.Thiswouldbetrueofanysuchlarge-scaleandhigh-levelsurvey,asitwillalwaysprovideonlyastartingpointforinvestigation.Itwillnotprovideallthedetailandnuancethatisrequiredregardingtheissuesthatitcovers,norwillitcoveralltheissuesthatarerelevanttoimprovingteachingandlearning.Asweshallseeinthenextsection,itneedstobeusedalongsidearangeofotherdatainordertoprovideameaningfulpictureofstudents’perceptions.Someinstitutionsfeelthataninsufficientappreciationofthiscanleadtomisdirectedeffort,withtimeandresourcesbeingfocusedonareasof

9 ForsomeoftheliteratureonthetechnicalpropertiesoftheNSS,pleaseseethefurtherreadinginSection5.3.

40

apparentconcern,whenfurtherinformationandcontextualisationwouldrevealthatthesupposedly‘poor’NSSscoresdonotreflectgenuineproblems.

Inadditiontotheseobstaclescausedorexacerbatedbyexternalpressures,thereareotherchallengesthatrelatemorecloselytothesurveytoolitself.Manyinstitutionshavenotedthatthesurveyitemsarenotsufficientlydetailedorexhaustivetocaptureeverythingtheywouldlike.Theitemsarebynecessitygenericandhighlevel,butthisalsoleavesgaps.Specificissuesraisedinclude:

• thelackofinformationaboutotherfactorsnotdirectlyrelatedtoteachingandlearning,suchascarparkingandlocalsurroundings;

• anabsenceofdetailabouttheparticularexperiencesofpart-timestudents;

• alackofdataaboutstudents’perceptionsofaspectsofthecourserelatedtoemployability.

Inaddition,onecontributinginstitutionalsofeltthatthesurveyisfocusedon‘traditional’three-year,singlesubject,face-to-facecourses,andsoisnotwell-suitedtoyieldinformationaboutstudentsonjointHonourscourses,studyingpart-timeoratadistance.TheseissuesallraisechallengesforthoseusingtheNSSforenhancement,andreinforcetheneedtosupplementthedatawithotherinformation,thetopicofSection4.2.

Therearesimilar,thoughlesswidespread,concernsaboutthegenericnatureofthequestionsrequestingtextcomments,whicharelookedatinSection4.4.Questionswhichasksstudentstohighlightpositiveandnegativeaspectsofthecoursemightbeimprovediftheyweremademorefocusedandincreasedinnumber.Oneinstitutionproposesthatquestionsaskingstudentstolistthreethingstheywouldchangeabouttheirprogramme,orincludinganoptiontoleaveacommentaftereverygroupofquestions,wouldimprovetheabilityofstafftounderstandthequantitativeresults.

Case study: Understanding student responses to the NSS – cognitive interviewing at Sheffield Hallam UniversityTheNationalStudentSurveyhasbeenvalidatedasareliablesurveyinstrumentandhasbeenresponsibleforagreatdealofenhancement.However,whattheresultsactuallymeanhasoftenbeenlessclear.Asonememberofstaffsaid“wehavelowresultsforthecourseisorganisedandrunningsmoothly,butwehavenoideawhatstudentsarereferringtowhentheysaythat”.Atourinstitution,alargeteaching-orientateduniversity,wedecidedtouseCognitiveInterviewingtobetterunderstandstudentresponses.Thistechniqueaimstouncoverhowsurveyitemsareinterpretedandwhatrespondentsthinkaboutwhenresponding.Themethodologyisverysimilartostructuredinterviewing;however,theaimistogetintervieweestoverbalisetheirthoughtprocessesastheycompletethesurvey.Apotofdevelopmentmoneywasupforgrabs,andwithapprovalforstafftimeandthefundsforincentives,wesetaboutrecruitingfinal-yearstudentsforone-to-oneinterviews.

Recruitmentforinterviewswasnoteasypost-NSS,aswewereheadingtowardsfinalexamsandprojecthand-ins.Ourfocuswastobeonindividualcoursesacrosstheinstitution.Thiswassothatwewouldhaveseveralgroupsofstudentswithsimilarlearningandteachingexperienceswithingroups,but

41

diverseexperiencesacrossgroups.Anaddedbenefitwasbeingabletoofferanin-depthpictureofthestudentexperienceoncoursestoraisestaffinterest.Theselectionofcourseswaslargelyonbestandworstperformers,agreedwithseniorfacultystaff.Recruitmentofstudentsvariedfromarandomselectionofstudentsonacourse,tolecturerssuggestingnamesofregularlyparticipatingstudents,tostudentsrecruitingfriends.Itwas‘whateverworks’recruitment.Thestudyendedupwithafairlyself-selectedgroupofstudents(nonedisagreedthattheyweresatisfiedwiththecourse)andself-selectedgroupofcourses.Gettingengagementfromstaffandstudentsatapressuredtimeoftheyearwassometimesimpossible.However,thecoursesdidincludeabroadrangeofsubjectsandasforthestudents,moreinamoment.

Threeinterviewerscarriedoutthe40one-to-oneinterviews.Basedonapreviousstudywedecidedtotakewrittennotesofverbalresponsesandofferedanopportunityforthestudenttotypereflections.Aftercomparingnotesandusingaverbalonlymethodforthefinalinterviews,itwasthoughtaudiorecordingswouldbebestinfuturetocapturethefulldepthofresponses.

Thefirstsurprisewasthewidediversityofstudentexperiencewithinthesamecourse,despitetheself-selectednatureofthesample.ThevariabilitycapturedbytheNSSreallydidreflecthowacoursecouldbeperfectforonestudentandyetbeviewedfarlesspositivelybyanother.Itevidencedthechallengetocoursedesigninsatisfyingdiversefactorssuchas:needforsupport,abilitytogainsupport,interestincontentandlifegoals.Despitethisdiversitystrongthemesemergedaroundthesurveyitems.Keyfindingswere:

• responsestotheNSScorrelatedwithknowngoodpracticeinlearningandteaching,includingaroundactivelearningandeffort;

• foranyoneitemmostrespondentsreferredtomultipleexperiences,whichcouldhavebothanegativeorpositiveinfluenceonresponses;

• forsomequestionstherewasabiggapbetweenstaffinterpretationofresponsesandwhatstudentswereactuallyreferringto;

• foratleastsomequestionstherewasvariabilityininterpretationacrosssubjectsthatimpactedonresponse;

• includedintheresponseswerestrongreflectionsofthestudent’sownengagementandabilitytoseekhelp,highlightingthattheresponseswereareflectionofbothcohortandstaff.

Sohowdoesaninstitutionusethisinformation?First,itcanhelpunpickwhatstudentsmightbereferringtowhenlookingatresults.Second,itcanhelpstaffgainanoverviewofhowstudentsperceiveissuessuchasfeedback.Third,ithelpsunderlinethatwhilesomeitemsdogiveavalidreflectionoflearningandteaching,theresponsesalsoreflectthediverseneedsandexperiencesofacohortofstudents.Theresultsfromthestudyarebeingmadeavailableonline,andwillcontributetoourknowledgeofstudentsurveys.However,itistheembeddingofthisknowledgeintheinstitutionalreportingofresultsandactionplansthathasperhapsbeenmostpleasing.Surveyresultshaveatendencytowardslistsofnumbersandboldsentences;thisresearchhashelpedblurtheedgesofthosestatementsandinforminterpretation.

Jason LemanHigher Education Academy (formerly Sheffield Hallam University)

42

References

Beatty,P.andWillis,G.2007Researchsynthesis:Thepracticeofcognitiveinterviewing.PublicOpinionQuarterly,71,287-311http://poq.oxfordjournals.org/content/71/2/287.short[accessed15October2012]

Gibbs,G.2010Dimensionsofquality.York:TheHigherEducationAcademy.http://www.heacademy.ac.uk/assets/documents/evidence_informed_practice/Dimensions_of_Quality.pdf[accessed15October2012]

Leman,J.2012StudentInterpretationoftheNSShttp://prezi.com/em7iec3qbxaa/student-interpretation-of-the-nss/[accessed15October2012]

Surridge,P.2009TheNationalStudentSurveythreeyearsonWhathavewelearned?York,TheHigherEducationAcademy.http://www.heacademy.ac.uk/assets/documents/research/surveys/nss/NSS_three_years_on_surridge_02.06.09.pdf[accessed15October2012]

AcommoncomplaintfrominstitutionsconcernsthelevelofaccesstheyreceivetotheNSSresults.Oneimportantelementoftheappropriateuseofthedataisdetailedandrigorousanalysis,butseveralinstitutionsfeelthatthisishamperedbytheincompletenatureofthedatatheyreceive.Forreasonsofconfidentiality,institutionsonlyreceiveaggregateddata,withthesmallestgroupbeingtenresponses.Thisisdesignedtopreventinstitutionsfromidentifyingindividualstudents,butasaconsequenceinstitutionsareunabletotrackstudentsthroughtheircourses.InstitutionsfeelthatbetteraccesswouldallowthemtoconnectNSSresultswithdataregardingexpectations,experiences,engagementandlearningoutcomes,andlearnagreatdealabouttheimpactofspecificteachingpracticesandstrategiesonawiderangeofdesirableoutcomes,includinglearninggainor‘valueadded’.

A key drawback of the NSS is the lack of access to the full dataset, even long after the student has finished their course. The ability to track individual students through institution would be welcome. The ability to scrutinise differences across populations within the institution would be beneficial to students.

Finally,someinstitutionsexpresssomeconcernsaboutthefocusoftheNSSonstudentsatisfaction:

• somemembersofstaffhaveprincipledobjectionstotheideathatstudentsatisfactionshouldreceivesuchprominentattention;

• oneinstitutionbelievesthatstudentsatisfactionisnotanappropriatemeasureofqualityinhighereducation,whentheunderlyingintentionistopreparethemforlifeafteruniversity;

• somestaffviewthefocusonsatisfactionascontributingtothekindofconsumerculturethatdamagestherelationshipbetweenstaffandstudents.

Student satisfaction is not a measure of quality in education. Many students might be satisfied – but that’s not really what it is about.

Oneinstitutionhasclaimedthatthislimitationcanbeoverestimated,asstaffarenotnecessarilyrequiredtofocusonsatisfactionjustbecausethesurveyis;theyarefreetolookatotheraspectsoflearning,suchasengagement.Thiswillbediscussedinthenextsection,butregardlessofwhatotherinformationisgathered,theNSS

43

willalwaysgivealevelofprominencetosatisfactionandstudents’positivityabouttheeducationtheyhavereceived.ThismaybesomethingtoaddressinfuturedevelopmentsoftheNSS.

Questions for reflection:

• DothebenefitsoftheNSSforqualityenhancementoutweighthedrawbacks?

• IstheNSSagoodmeasureofeducationalquality?Whatothertypesofinformationprovideacomplementaryorsuperiormeasure?

• DoestheNSScontributetoadamagingconsumerculture,orcanitbeharnessedtoimprovethepedagogicrelationship?

• HowdoyouuseyourunderstandingofthelimitationsoftheNSStomakemoreeffectiveuseofitforenhancement?

44

4.2 Triangulation with other data

Key points

• Togainamoreroundedandusefulunderstandingofstudents’experiences,andtoinvestigatethereasonsforthescoresandwaystorespond,institutionshavetouseNSSdatainconjunctionwithamuchwiderrangeofinformation.

• Forsomeissues,supplementarydataarerequiredinordertocorroborateandunpackNSSscores.Forotherissues,theNSSmaybeentirelysilent.

• Someinstitutionsareexploringtheuseofengagementsurveys,aswaysofunderstandingwhatstudentsareputtingintotheircourses,ratherthanwhatinstitutionsareproviding.

TheprevioussectionbrieflydescribedhowtheuseoftheNSSinisolationcanseriouslyhamperitsuseasatoolforenhancement.Institutionsarewellawarethatitsrationaleanddesignmeansthatitcanonlyprovideaveryhigh-level,coarse-grainedoverviewofstudents’experiencesoftheircourse:

• beingintendedforuseinallinstitutionsandallsubjects,thequestionslackthekindofcontext-specificfocusthatisessentialtogainingatruepictureofthoseexperiences;

• beingdesignedwithaviewtoprovidingahighresponserate,thequestionsarelimitedinnumberandcomplexity;

• andbeingcreatedprimarilyfortheproductionofinformationthatcanbeusefultoprospectivestudentsinmakingcoursechoices,thesurveyhasaparticularfocusthatisnotalwaysidealforenhancementefforts.

ThesefactorsmeanthattheNSS–likeanysinglesourceofinformation–isonlypartofthepictureandshouldbeusedasonetoolalongsideothers.

NSS data always needs to be reviewed in the context of other information – it is becoming increasingly apparent that vagaries in the interpretation of the data (i.e. jumping to conclusions on the basis of NSS data alone) can lead institutions down wrong, often resource-intensive paths.

ThereisavirtuallyuniversalacknowledgementthattheNSSshouldbeusedaspartofaholisticprocessofgatheringandactingondataaboutlearningandteaching.InmanycasesthisisexpressedastheideathattheNSSshouldfunctionasastartingpointforfurtherinvestigation;asanindicationofareasthatwouldbenefitfromfurtherexploration,ratherthandefinitivedataaboutsuccessesandfailures.SeveralinstitutionsmakethepointthattheNSSresultsontheirownwillnotprovidemuchinformationonthespecificsofanyproblems,thereasonslyingbehindhighorlowscores,orwhatactionscouldbetakeninresponse.

Section4.4looksathowqualitativedata,gatheredthroughtheNSSandbyothermeans,canilluminateandsupplementthenumericalNSSresults.Thissectionwillfocusonthewideruseofquantitativedata.

TheneedforawiderrangeofinformationisparticularlypressingforareaswheretheNSSdoesnotyieldrelevantdata.Institutionshavecitedthefollowingexamplesofissuesthat‘fallthroughthegaps’intheNSS:

• thereisalackofinformationaboutthewiderstudentexperience,suchascar-parkingandlocalclimate;

• whiletheNSSisusefulforemployabilityissues,thereisaneedtogathermorefocuseddata;

• aspectsofpart-timelearningarenotaddressedinthesurvey,andprogressionandretentiondatamaybeparticularlyrelevant;

45

• duetothewaythattheirresponsesareallocatedtosubjects,thecollectionoftheviewsofjointhonoursstudentsrequiresfurtherwork;

• forinternationalstudents,someinstitutionsfeeltheneedtoundertakemorebespokework,andtheInternationalStudentBarometerhasbeenfounduseful(thoughnotuniversallyso).

Case Study: Exploring assessment and feedback at the University of WinchesterWithinthesub-culturesandmicro-culturesofinstitutionsandprogrammesrespectively,thetriangulationofdata(particularlyqualitativeandquantitativedata)oftenallowsamorethoroughunderstandingofthephenomenonbeinginvestigated,whichcontributestothe,sometimes,abstractnumbers.InevaluatingthepoweroftheNSS,oneoftheregularcomplaintsisthatitdoesnotinformprogrammesastowhythescoresthatyearwerehigherorlowerthanthoseofthepreviousyear.Thishasremainedacommonboneofcontentiongiventhatleaguetablesutilisingthedatadonottypicallygeneratemeasuredchanges,butoftenmoreknee-jerkreactionswithoutfullyunderstandingtheissues.

AlthoughtheNSSprovidesavaluableperceptionfromthestudentcohortsastotheareasof(dis)satisfaction,thereislittleinformation,dataordirectionastowhatcouldbedonetoenhancethisaspectofthestudentlearningagainsttheirexpectationsortofullyinterrogatethestudents’perceptionofit.TESTA,thethree-year,HEA-funded,nationalproject,workedacrosssevenprogrammesinfouruniversitiesandemployedamethodologythatspoketotheneedfor:usefuldataaroundassessmentandfeedbackexperiences,datatobehighlyrelatedtotheprogramme,qualitativedataindescribingtheexperienceofstudents,collaborativeevaluationandreflection.ThisbuiltfromtheNSS,inusingadditionalsourcesofdatathatcouldbettertriangulatetheaspectsoftheassessmentandfeedbackenvironment,fromwhichenhancementactivitycouldthenfollow.ThedatacollectedwithinTESTAincluded:theAssessmentExperienceQuestionnaire(AEQ),anauditofeachprogrammes’assessmentenvironmentandfocusgroupdatafromstudentswithintheirthirdyearofstudyacrossthesevenprogrammes.

Caseprofileswerethenwrittenandpresentedbacktothewholecourseteams,whereaparticularlycollegiateapproachwastakentoensurethatstaffdidnotfeeltheywerebeingtoldwhattodo,butinstead,werebeingallowedtoexplorethedataanddebatethefindingsthroughtheirownlensesandasateam.Thetriangulateddataallowedtheprogrammeteamstorelatetothetypesofdatathatmostspoketothemandtoviewtheaudit,AEQandfocusgroupdatatogether.Similarlythespaceprovidedallowedarichdiscussionaroundsomeoftheunderlyingprinciples,someofwhichwereestablished20or30years.Thisallowedstafftotryandidentifykeypracticalsolutionstotheassessmentandfeedbackenvironment/practiceinawaythatsawsomequitesignificantchangesdevelopedandimplementedbytheprogrammeteams(withnostickorcarrotinsight).

GiventherelativelyslownatureofchangeacrossinstitutionscoulditbethatoneshouldexpectthesamelevelofslowevolutionwithassessmentandfeedbackasillustratedfromtheNSSscoresovertime–Ithinknot.Coulditinsteadbethatmultiplesourcesofdatacangenerateevaluation,reflectionandchangefromprogrammesteams,iftheyareaffordedthetime,spaceandresourcetotrulyreflectandenhanceprogrammes,basedonusefulevidence

46

andincreasedawareness/understandingoftheirstudents’assessmentandfeedbackexperiences?IthinktheNSSaccuratelyfocusesattention,butitisincumbentuponinstitutionstofullyinterrogatetheelementsutilisingadditionaldata,beforechangingthingsonawhimorahunch. Yaz El HakimUniversity of Winchester

Institutionsoftenclaimthattheyuseaprocessof‘triangulation’,butwhatdoesthatmean?Inbasicterms,thisreferstotheprocessofcomparingNSSresultswithothersourcesofinformationinordertocreateamoreaccuratepicture,amoredetailedpicture,oramorecomprehensivepicture.Therangeofothersourcesofinformationusedbyinstitutionsinthisprocessisvast,andthefollowingisjustaverypartiallistofexamples:retentionstatistics,employabilitydata,degreeclassifications,modulegrades,internalinstitutionalsurveyresults,moduleevaluationquestionnairedata,equalitydata,externalexaminerreports,andwithdrawalsurveys.Mosttriangulationprocessesaresimple,butatsomeinstitutionsthoseinvestigationshaveledontocomplexexternallyfundedresearchprojects.

NSS data is only ever one motivating factor in change. The overarching approach is to triangulate issues arising from the NSS with other data and analysis, and make changes in policy and practice in the light of this full analysis.

Manyinstitutionshighlighttheimportanceofpayingattentiontoalternativesourcesofstudentfeedback.TheNSSreceivessuchavastamountofattentionthattheremaybeatendencytooverlookorunderuseotherwaysinwhichstudentscanprovideinformationabouttheirexperiences;wayswhichmightbettersuitcertainpurposesormightbetteraddressparticularissues.Acommonalternativesourceoffeedbackisinternalinstitutionalsurveys,whichhaveexistedformanyyearsasessentialsourcesofinformationfortheevaluationandimprovementofteaching.EffortstocontextualiseandunderstandNSSscorescanbegreatlyassistedbyhavingresultsonthoseverysamequestionsfromstudentsearlierintheircoursesofstudy,andseveralinstitutionshaveadoptedNSSquestionsfortheirfirst-andsecond-yearsurveys.However,thereisalsoanawarenessthatinstitutionalsurveysofferanopportunitytogainmoredetailedandcontext-specificinformationfromtheirstudents,thatismissedifNSSquestionsareused.

Therehasbeenalotoftalkrecentlyaboutthepossibilityofgaininginformationaboutthelevelandqualityoftheeffortthatstudentsputintotheircourses.TheNSSlargelyfocusesontheothersideoftheprocess,theeducationalprovision,andsosomeinstitutionshavestartedtolookelsewhere.Severalhaveimplementedengagementsurveys,basedontheNorthAmericanNSSE(NationalSurveyofStudentEngagement)10anditsAustralasianvariant,theAUSSE(AustralasianSurveyofStudentEngagement)11.Thosesurveysaskstudentsabouttheirbehaviouranddevelopment,focusedaroundthetimespentpreparingforclass,theamountofengagementwiththecoursematerialandwithpeersandteachers,andthedepthoftheirlearning.Institutionsbelievethatthesesurveysallowthemtogathermoredetailaboutthelearningenvironment,andascertainwhethercourseleaders’intentionsareborneoutbythestudents’behaviour.

More information is needed in order to be fully informed, and the NSSE and AUSSE do look at engagement, which is also an omission – I can be very satisfied with my gym membership and facilities but it doesn’t mean I have used them very well or very often.

10 http://nsse.iub.edu/[accessed15October2012].11 http://www.acer.edu.au[accessed15October2012].

47

TheoverwhelmingbenefitsofusingtheNSSaspartofarangeofinformationarewidelyacknowledged,butitalsobringschallenges:

• therearesignificantpracticalissuesinvolvedinmappingindividualsandsubjectgroupsacrossdifferentsurveysanddatasets;

• oneinstitutionfeelsthatthemessageemergingfromtheNSSbecomesblurredandconfused,andconcreteactionsbecomemoredifficulttojustify;

• anotherinstitutionworriesthatthedefensiveattitudeofsomestafftowardstheNSScanleadthem,whenfacedwitharangeofinformation,to‘cherry-pick’thedatathatpresentstheiractivitiesinthebestlight.

Theseareallissuestobeawareof,buttheyhavenotdissuadedanyoftheconcernedinstitutionsfromusingtheNSSaspartofawidersetofinformation.

TheintentiontousetheNSSalongsideothersourcesofinformationisvirtuallyuniversalamonginstitutions.Thetendencyofthemediatoseeitasanaccuratesummativemeasureofteachingqualitycancreatepressurestouseitinisolation,butinstitutionsaretypicallywellawareoftheneedtolookbeyondtheNSSinordertounderstandtheirstudents’experiences.

Questions for reflection:

• IstheNSSusedasastand-alonemeasureofteachingquality,orittreatedasonetoolamongothers?

• Whatinformationaboutlearningandteachingisrequiredtoinformenhancementactivities?HowdoestheNSSfitintothat?

• IftheNSSjusttellsyouwheretolook,whatisthebestofwayofexploringfurther?Whatinformationwouldhelpyoudecidehowtorespond?

48

4.3 Benchmarking

Key points

• Institutionsuniversallyattempttomakelike-for-likecomparisonswhenusingtheirscores,andtherearearangeofmethodsadopted.

• Comparisonsbetweenitemscoresshouldonlybemadecautiously,becauseofthedifferentconceptsinoperation.

• RankingsandleaguetablesusingNSSscoresareonlycrudemethodsofbenchmarking,astheydonottakeintoaccountimportantinstitutionalcharacteristics.

• BenchmarkingisanimportantelementoftheNSScycle,butitisalsoimportantnottogetboggeddown;whatiscrucialisexploringissuesfurtherusingotherinformationsources,andundertakingenhancementactivitiesinresponse.

ManyinstitutionsfeelthattheabilitytocomparedataonstudentperspectivesisoneofthekeybenefitsoftheNSS.Thecompulsoryanduniversalnatureofthesurvey,andthepublicationoftheresults,meansthattheNSSisanunprecedentedtoolforbenchmarkingstudents’perceptionsatinstitutionandsubjectlevel.

It is very useful for people to have an idea of where they sit relative to other institutions and for people to be prompted to think about what they do

Institutionsexpressedarangeofmorespecificpoints,suchasthefeelingthatbenchmarkingopportunitieswerelimitedbeforetheintroductionoftheNSS.Itwasalsostatedthattheavailabilityofnationallycomparabledatahasledtogreaterattentiontothestudentexperienceandthestudentvoice,andthatthedesignoftheNSSmeansthatitprovidesneatandsimplecomparisons.

Therearevariousapproachestocomparisonsandbenchmarking,butallinstitutionsattempttocompare‘like-with-like’.Thiscansimplymeancomparisonswiththemissiongroupthattheinstitutionbelongsto,oritcanmeancomparisonsagainstabespokegroupofinstitutions,felttorepresentinstitutionswithrelevantlysimilarcharacteristics.Forexample,aScottishinstitutionreportsthatitcompareswiththerestofScotland,asthatisfelttobetherelevantpeergroup.Thiskindofbenchmarkingisfelttobeimportantforascertainingwhetheraconcerningscorerepresentsagenuineproblem,orwhetherthatscoreisactuallywithinappropriateexpectations,giventhenatureoftheinstitution.

Case study: Benchmarking at the University of HertfordshireInconstructingbenchmarksweusethreedifferentcategorisations;JACSCodeLevel3,HESACoursesand40DepartmentIDcodes(comprisedofgroupsofdiscipline-relatedprogrammeswithinaSchool).

TheperformancebyDepartmentIDandHESACourseoneachsectionandeachquestionofthesurveyisputintoazone,dependingonhowthatareaisperformingrelativetothemeanforthatNSSsectionorquestion.Thezonesareidentifiedandcolourcodedasfollows:

49

Zone 1 Morethantwostandarddeviationsbelowthemean (seriouslyunderperforming)

Zone 2 BetweenoneandtwostandarddeviationsbelowthemeanZone 3 UptoonestandarddeviationbelowthemeanZone 4 UptoonestandarddeviationabovethemeanZone 5 BetweenoneandtwostandarddeviationsabovethemeanZone 6 Morethantwostandarddeviationsabovethemean

(exceptionalperformance) AsimilarprocessisusedwiththeJACSCodeLevel3datatoascertaintheperformanceofdisciplinesrelativetodisciplinaryperformanceacrossthesector.Threeseparatecolour-codedspreadsheetsareproduced;oneforeachofthethreedifferentcategorisations.

Thebenchmarkdataareusedtoidentifyandcelebrateexcellenceaswellasforidentifyingprogrammesofconcern.Additionally,itisusedintargetsettingforSchools.However,thebenchmarksalsoplayakeyroleinpromotingenhancementthroughthedevelopmentofSchoolActionPlans.WhereresultsforaparticularsectionofthesurveyfallintoZones1-3ofthe‘trafficlight’system,Schoolsareaskedtogenerateanaction.

ActionplanningisfacilitatedthroughtheholdingofahalfdayNSSForum,explicitlydesignedtobebothsharingandsupporting,whichbringstogetherareasthatare‘underperforming’alongwiththosewithhighsatisfactionscores.TheNSSForumisattendedbyseniormanagersfromallSchools(e.g.thoseresponsibleforprovidinglearningresources),seniorstafffromcentralunits(e.g.thosechargedwithsupportingtheenhancementofthestudentlearningexperience),studentrepresentativesfromeachSchoolandStudents’UnionSabbaticalOfficers.Thesessionstartswithashortpresentation,inwhichUniversity-levelquantitativedataareconsideredalongsidethethemesthathaveemergedfromthecommentsmadebystudentsaspartoftheirNSSresponse.Next,usingtheirknowledgeofthedataandoflearningandteachingactivitieswithintheSchools,stafffromtheLearningandTeachingInstitutegroupSchoolstogethertoworkonareaswhereactionisrequired.Schoolswithhighlevelsofsatisfactioninaparticularareaworkwiththosewithlowerlevelsofsatisfaction.Thestudentshelptocontextualisethecommentsandimportantly,workwiththestafftodevelopaSchoolActionPlan.TheActionPlans,whichformpartoftheAnnualSchoolReport,includeacommunicationstrategydesignedtoensureeffectivefeedbacktostudentsonhowaSchoolplanstoaddressidentifiedchallenges.

Recommendations for others

Asstaffareveryawareofthevariabilityofsatisfactionscoresfromonedisciplinetoanother,ithasbeenimportanttoreflectthesedifferencesinthebenchmarkingprocess.Byprovidinganenhancementfocus,asupportiveenvironment,andaneffectivemechanismforsharinggoodpractice,theNSSForumhasgonesomewaytoalleviatingstaffconcernsregardingthepotentialofthebenchmarkingprocesstofocusonfailureandunderperformance.

Catherine RendellUniversity of Hertfordshire

50

Comparisonsatsubjectlevelcanbeparticularlyuseful,especiallyconsideringtheimpactofsubjectofstudyonstudents’learningexperiences12.Someinstitutionsbenchmarktheirdepartmentsandcoursesinternally,againsteachotherandtheinstitutionalaverage.Moreinstitutions,however,comparedepartmentsandcoursesagainstsimilardepartmentsandcoursesinotherinstitutions,togetamoreaccuratesenseofwherethereareareasofweaknessandstrength.

Oneinstitutionbelievesthatitisimportanttobeawarethateachitempromptsdifferentresponses,somereferringtoquitespecificexperiences,otherstoabroadrange.Ifanitemreceivesalowscorerelativetootheritems,thatdoesnotautomaticallyindicateapriorityproblem,orarelativelyhighscoreindicatethatthereisnoissue.Itiscrucialtorecognisethatitemscoresarebestcomparedagainsttheitemscoresforthesectororforspecificgroupsofpeers.Giventhedifferentconceptsinoperationforthedifferentitems,comparisonsbetweenitemscoresshouldonlybemadecautiously,especiallyforitemsindifferentquestiongroups.

Ofcourse,oneoftheprimarymethodsofcomparinginstitutionalandcoursescores,andcertainlythemostvisibletothepublic,isthroughrankingsandleaguetablesinthemedia,andthewebsitesthatcomparethenewKeyInformationSets.ThesecomparisonscertainlyplayakeyroleinhowtheNSSisperceivedandusedwithininstitutions,andseveralinstitutionshighlightedtheimpactthatpositioninleaguetablescanhaveontheperspectivethatistaken,especiallybyseniormanagers.Oneinstitutionevencommentedontheeffectonstaffattitudesofachangeinthealgorithmusedbyaprominentnewspaperintheconstructionoftheirleaguetables,givingmoreweighttotheNSS.

Severalinstitutionshighlightedtheproblemthatthiscauses.Theycommentonthefactthatleaguetablescompareinstitutionsusingtheirrawscores,withoutregardtothefactorsthataffectthosescores,suchassubjectmix,demographicprofileandsize.Theimportantelementofbenchmarkingiscomparinglike-with-like,somethingwhichthemediastandardlyoverlook.

The way the press use it for league tables bothers me, we are not comparing like with like as institution mission etc. is not taken into account.

OneinstitutionfeltthattherewouldbebenefitinHEFCEprovidingmoreforcefulguidancetothemediaonhowthesurveydatashouldandshouldnotbemadeuseof.HEFCEiscertainlyawareoftheconcerns,andattemptedtorespondby,from2011onwards,producingbenchmarkedresultsforoverallsatisfaction,whichtakeintoaccountimportantinstitutionalfactorssuchassubjectandgendermix.NotonlydoesthiscommunicatethesubtletiesinvolvedininterpretingNSSresults,italsoaddressestheinstitutionalconcernthatthemediadonotcomparelike-with-like.However,oneinstitutionstatedthatthebenchmarkscoresaretoocomplexandhardtounderstandtohavearealimpactontheactivitiesofthemedia.TheysuggestedthatinsteadHEFCEfacilitateaprocesswherebybenchmarkingclubscanbecreated,includinginstitutionswithsimilarcharacteristics.

Oneinstitutionstatedthattheywerestartingtolookbeyondthetraditionalapproachtobenchmarking,andinvestigatethestudentperspective.Whatdoesastudentthinkthata‘good’scoreis?Isthatthesameaswhatappearstobea‘good’scoreinsectorcomparisons?

Finally,therewasasensethatitisimportantnottogetboggeddownintheprocessofbenchmarkingandcomparison.WhileitisanimportantstageintheNSScycle,itiscrucialtousetheresultsasstartingpointsforinvestigation,andtomoveontothekeystepofundertakingenhancementactivitiesinresponse.

12 Formoreontheimpactofsubject,seetheliteratureintheSection5.3.

51

Questions for reflection:

• Whatspecificinformationareyoutryingtogainthroughtheprocessofmakingcomparisons?

• Arebenchmarkingexercisesdesignedtoservemarketingandrecruitment,orqualityenhancement?

• Doyoucomparedepartmentsandcourseswitheachotherwithintheuniversity,orwithsimilardepartmentsandcoursesaroundthesector?

• DoesleaguetablepositionhaveanimpactontheresponsetotheNSS,regardlessofwhethertheleaguetablemethodologyisaccepted?

52

4.4 Using qualitative data

Key points

• InstitutionsusethecommentsfromtheNSStoexplorethereasonsbehindthepatternsinthequantitativeresults.

• Institutionsfindthecommentstobemoreeffectiveatengagingstaffindiscussionsaroundstudentfeedback.

• Theuseofstudentandstafffocusgroups,especiallyatdepartmentalandcourselevel,allowmuchdeeperexplorationofstudents’experiencesoftheircourse.

InSection4.2wesawhowinstitutionstendtousetheNSSalongsideothersourcesofinformation,allowingthemtoexploreissuesinmoredetailandworkouthowtorespond.Formanyinstitutions,animportantelementofthatprocessistheuseofqualitativedata.Therearemanywaystogatherqualitativedataaboutstudents’experiences,rangingfrominformalconversationsincorridorsthroughtolarge-scaleresearchprojects.Oneready-madesetofdataexistsinthecommentsthatstudentscansubmitaspartoftheirresponsetotheNSS,whichhavethebenefitofbeingcompletedbythesamepeople,atthesametime,andwiththesamefocus;theircourse-levelexperiences.Theinvestmentofsignificantamountsofenergyintheuseofthefree-textdata,especiallyinrelationtodataanalysis,seemstobearelativelyrecentandgrowingphenomenon.Moreandmoreinstitutionsappeartobeincorporatingthecoding,analysisandreportingofthestudents’commentsintothesurveycycle,alongsidetheroutineanalysisandreportingofthenumericalscores.

InstitutionsreportarangeofmotivationsforexploringtheNSScomments,withthemostcommonbeingtoshedlightonthenumericalresults.Thereasonsforacourse’slowscoremaybeentirelyinvisibleinthequantitativedata,butanysimpleandpowerfulconcernsthatthestudentshavearelikelytoemergeintheircomments.Forstaffseekingtomakesenseofoftensurprisingresults,thecommentshaveproveduseful.Inaddition,thecommentscansometimessurfaceissues,suchasvalueformoneyandemployability,thatarenotdirectlyaddressedbytheNSS.Thecommentsprovideaholisticpictureofstudents’experiencesthatnumericalresponsesareunlikelytocapture.

The staff find the open comments very helpful in trying to make sense of the figures the survey produces.

Anothermotivationcitedbyinstitutionsisthepositiveresponsefromacademicstaff.Whereassomecolleaguesmaybedisengagedbynumericalscores,oradoptdefensiveattitudes,thereisasensefromsomeinstitutionsthatthecommentsaremoreeffectiveatgeneratinginterestanddiscussion.Theemotiveanddirectnatureofstudents’commentsseemstoaidtheuseofthesurveyasapromptforreflectionandenhancement;theyareseenasthe‘softerside’oftheNSS.Oneinstitutionbelievesthatthehigherlevelofengagementwhenthequalitativedataarethefocusitpartlyduetothefactthatitmovesdebateawayfromtherobustnessofthequantitativedatatowardsdiscussionofthefundamentallearningandteachingissues.

More recently we have been using the free text comments in a more developmental way.

53

Case study: Semantic analysis of free-text comments at Liverpool John Moores University What we did

Werecognisedthatthequalitativefree-textcommentsoftheNSStendtobeunderutilisedbuthadthepotentialtoprovideavaluablesourceofinstitutionalfeedback.AtLJMUwealsoconductourown‘mirror’surveysatYears1and2,providingopportunitytocontrastthestudentexperienceacrossthreelevelsofstudy.However,collectivelythesesurveysprovidec.150,000wordsofqualitativecomment,whichisnosmallchallengetointerpret. Weanalysedfree-textdatafromNSSandLJMUstudentsurvey(identicaltoNSSforYears1and2)usingsemanticanalysissoftwareLeximancer(https://www.leximancer.com).Thesoftwareidentifiestextualstructureofstudentcommentsbyrevealingkeyconceptsandbroaderoverarchingthemes(conceptclusters).Conceptmapsgeneratedbythesoftwareidentifythemostimportantareasofexperience,surfaceinter-relationshipsbetweentheseconceptsandidentifytheirassociatedsentiment,thestatisticallikelihoodofstudentsreferringtotheconceptinfavourableorunfavourableterms.

Thisinstitutional-levelanalysiswascarriedoutovertwoyears,enablingustoidentifychangesinthe‘studentsatisfactionlandscape’,includingappearanceofnewconcepts,dynamicsofsentimentsandthematictransformations.Thefindingssurfacedsignificantdifferencesinthestudentprioritiesandattitudesacrossthelevelsofstudy.Thelongitudinalanalysisdemonstratedtheimpactofmajorinstitutionalinitiatives/interventionsandorganisationalfactorsonstudentsatisfactionanditsrepresentationontheconceptmap.Theoutcomesoftheanalysiswerewidelyusedinvariousacademicenhancement/developmentsessionsacrossacademicfacultiesandprofessionalserviceteams.

Why we did it

Thefree-textcommentsconstitutearichsourceofstudentfeedbackwiththepotentialtonotonlyilluminatescores,buttoidentifysignificantthemesthatfallthroughthegapsofthesurveycategories.InstitutionallevelanalysisofsuchacopiousdatasetasNSSfree-textcommentsistime-andresource-consuming,butcouldbeofgreatvaluetoinstitutionalpolicymakersandacademicdevelopersastheyidentify‘meta-themes’thatcrosssubject/facultyboundaries.That’swhyautomatedsemanticanalysiswaspilotedinitially.

Whether it worked

Thisapproachtoanalysisprovedtobesuccessful.Lookingattheconceptmapslongitudinallyhelpedtodetectchangesinstudentattitudes,perceptionsandunderstandingsandpresentedadynamicinstitutionalviewofthestudentexperience,intheirownwords.

Recommendations for others trying it out

AlthoughLeximancervisiblyreducestimeoftheanalysis,theunderlyingmeaning(s)ofaspecificconceptcanonlybeidentifiedthroughmanualinterrogationofthecomments.Sopreparetodosomeintensereadingandsensemaking.

54

Beawarethattherearesomelimitationstowhatthesoftwarecanprovide.Someconceptsrepresentedbynarrowvocabulary(e.g.‘lecture’or‘library’)favourastrongrepresentation.Whileothers(suchas‘workbasedlearning’),haveagreaterlikelihoodofbeingdilutedontheconceptmap(e.g.betweenplacements,fieldtrips,‘life’projectsandsoon).Aseparate,tailoredanalysiscouldbedonetoexplorethese‘diluted’concepts.Triangulationwithotherdatasourcesisneededtocreateamorecomprehensivepictureofthestudentexperienceandsatisfaction.

Elena Zaitseva, Clare Milsom and Martyn StewartAcademic Enhancement Unit, Liverpool John Moores University

SomeinstitutionsdoreportcomplexitiesandsensitivitiesaroundtheuseoftheNSScomments:

• oneinstitutionreportsthattheyhavenotfoundtheeffortrequired–whichisoftenconsiderable–tobeworthwhile;

• anotherfeelsthatthecommentsofthestudentstendtobefairlygeneric,andareonlyusefulwhenthereareoneortwoveryspecificissuesaffectingalargeproportionofthestudents;

• oneinstitutionbelievesthatwhiletheanalysisofthecommentsishighlybeneficialtoenhancementactivities,itneedstobedonebyexpertresearchersinordertoyieldrobustdatathatcanbereliedupon.However,theysaythatwhilethisisresource-intensive,itisbettertoundertakethatanalysisinasophisticatedmannerthantoputeffortintogatheringmoredatafromothersources;

• anotherisconcernedthatwhilethecommentsthemselvesmaybemoreimmediatelyengagingforstaff,theylacktheleveragepossessedbythenumericalscores,duetoinstitutionalpressuretorespond.

Ithasbeenobservedbytwoinstitutionsthatthoughuseful,thecommentsarestillrelativelysuperficial.Theyareusuallybrief,andoftentooindividualtopaintageneralpicture,andtheycannotbythemselvesprovideareliablepictureofthestudentlearningexperience.Whatislikelytobeneeded,asdiscussedinChapter2,isadialoguewithstudents,necessitatingother,morein-depth,waysofgatheringqualitativedata

• oneinstitutionrecruitsnewfirst-yearstudentstocreatephotodiariesovertheinitialmonth,photographingandcommentingonthingsthatstrikethemasparticularlygoodorbad;

• anotherinstitutiontracksfiveindividualstudentsthroughtheiruniversitycareers,gatheringdetailedinformationabouttheirexperiencesastheyprogress.

SomeinstitutionshavefoundsuccessinusingtheNSScommentsaspromptsfordiscussioninfocusgroups.Byaskingparticipantstocommenton,tounpackandtoexploretheimplicationsofrepresentativecomments,institutionshavefoundanapparentlyeffectivewayofgettingricherinformation.Moregenerally,focusgroupsareacommonmethodthatinstitutionsusetoinvestigateNSSresults,withorwithouttheuseofthequalitativedata.

Faculties also use focus groups to explore particular issues.

55

Aparticularlycommonstrategyistousefocusgroupswithincoursesanddepartmentsthathaveachievedeitherveryhighscores,orverylowscores.Someinstitutionshavealsorunparallelfocusgroupswithstaffandstudents,inordertodiscoversimilaritiesanddiscrepanciesandprovokereflectiononthepartofthestaff.Onetechniqueusedistoaskstafftocompletethesurveythemselves,fromwhattheyfeeltobethepointofviewoftheirstudents,andthentocomparetheiranswerswiththeactualscoresreceived.Discussionofthesedatawithinthefocusgroupscaninitiateadiscussionabouthowstudentsareperceivingtheircourses,andareasinwhichstaffandstudentshavedivergentperceptions.

Whateversourceinstitutionsutilise,andhowevertheythenputthemtouse,qualitativedataseemtobeaneffectiveoptionforgoingbeyondthenumericalscoresinordertogainaricherunderstanding,andengagingstaff(andstudents)intheprocessofenhancement.

Questions for reflection:

• Whatistherightbalanceofformalandinformalmethodsforgatheringqualitativedataabouttheexperiencesofstudents?

• Howshouldstudentcommentsbeusedinstaffmeetingsandfocusgroups,inawaythatmaximisesengagementanddiscussionbutissensitivetotheemotivenatureofstudents’comments?

56

5. Summing up and going further

5.1 Suggestions for good practice

Thefollowingarethekeypointsfromthroughoutthereport.AsstatedintheIntroduction,theyareintendedtobeusefulpromptsforconsiderationandreflection,ratherthanuniversalrecommendations.

Staff-student partnerships

1 NSSresultsneedtobeconnectedtothewiderprocessofstudentrepresentation,toensurethatthesurveyispartofthewiderconversationsbetweeninstitutionsandstudents.

2 IfstudentsaregoingtoplayanactivepartinthediscussionofNSSresultsincommitteesandworkinggroups,thentheyneedsupport,trainingandfullaccesstotherelevantresourcesanddocuments.

3 Partnershipsbetweeninstitutionsandstudents’unionsareoneofmostimportantelementsofinvolvingstudentsintheNSS,butboththeopportunitiesandthechallengesshouldbeopenlyacknowledgedanddiscussed.

4 Withsupportandadvice,studentscanplayanactiveroleintheanalysisofNSSdata,whetheraspaidresearchers,throughtheSUorthroughtheirstudies.

5 Jointstaff-studentevents,suchasworkshopsandconferences,canprovideaninformalbuteffectivemethodofinvolvingstudentsintheexplorationofNSSresults.

6 SUsareentitledtoreceivefullaccesstothedata,whichcreatesexcellentopportunitiesforstaff-studentcollaborationsondataanalysisandinvestigation.

7 FeedingbacktostudentsontheactionstakeninresponsetotheNSSisakeypartofengagingthemwiththesurvey.

8 ‘Yousaid…wedid’providesaverydirectmethodoffeedback,buttheremaybealternativewordingsthatmorestronglyemphasisepartnershipanddialogueratherthanrespondingtoconsumerdemands.

9 Ifthesurveydataarebeingusedappropriately,aspartofabroadersetofinformation,itmaybehardtolocatechangesmadespecificallyinresponsetotheNSS.Inthiscase,themessagethatisfedbacktostudentsmaybemoresubtleandcomplex.

57

Institutional structures

10 ThecommonperceptionthattheNSSisa‘sticktobeatuswith’canbemitigatedbycapturingandcommunicatinggoodpracticewhereresultsarebetterthanexpected,andnotjustfocusingontheareaswithdisappointingsurveyscores.

11 JustasinvolvingstudentsintheNSScanbebeneficial,engagingstaffatalllevelsintheprocessofpromoting,exploringandrespondingtotheNSSisakeypartofusingthesurveyforenhancement.

12 TheNSScanoftenbeapowerfulbutunspokenpresence;therearebenefitsinbringingitoutintotheopensothatitsstrengths,limitationsandimpactcanbehonestlydiscussed.

13 IntegratingtheNSSintoplanningandreviewcyclescanpromoteamorereflectivequalityenhancementapproach.

14 WheninstitutionsusetheNSStocreatetargetsandKPIs,itisimportanttobearinmindthelimitationsofthedata,andtheneedtotriangulateNSSscoreswithothersourcesofinformation.

15 TheNSShashelpedtocreateanewfocusonthestudentlearningexperience.

16 ThesimplisticuseoftheNSSinthemediaisadouble-edgedsword.Whileitcausesresentmentandsometimesanarrowfocusonraisingscores,italsocreatesaleverforchange.

17 IndividualsworkingtoenhancelearningandteachingcanuseNSSresultstopersuadecolleaguesoftheneedforchange.

Analysis and exploration

18 ItiscrucialthatthereisawarenessandhonestyaboutthelimitationsoftheNSSasatoolforenhancement.

19 TheNSSonlyprovidesapartialpictureofstudents’experiences,andneedstobesupplementedinordertobeuseful.

20 TheNSSgivesprominencetostudentsatisfaction,butitisimportantthatinstitutionsgatherdataonotheraspectsoflearning,inordertohavearoundedviewofeducationalquality.

21 Togainamoreroundedandusefulunderstandingofstudents’experiences,andtoinvestigatethereasonsforthescoresandwaystorespond,institutionshavetouseNSSdatainconjunctionwithamuchwiderrangeofinformation.

22 Forsomeissues,supplementarydataarerequiredinordertocorroborateandunpackNSSscores.Forotherissues,theNSSmaybeentirelysilent.

23 Someinstitutionsareexploringtheuseofengagementsurveys,aswaysofunderstandingwhatstudentsareputtingintotheircourses,ratherthanwhatinstitutionsareproviding.

58

24 Institutionsuniversallyattempttomakelike-for-likecomparisonswhenusingtheirscores,andtherearearangeofmethodsadopted.

25 Comparisonsbetweenitemscoresshouldonlybemadecautiously,becauseofthedifferentconceptsinoperation.

26 RankingsandleaguetablesusingNSSscoresareonlycrudemethodsofbenchmarking,astheydonottakeintoaccountimportantinstitutionalcharacteristics.

27 BenchmarkingisanimportantelementoftheNSScycle,butitisalsoimportantnottogetboggeddown;whatiscrucialisexploringissuesfurtherusingotherinformationsources,andundertakingenhancementactivitiesinresponse.

28 InstitutionsusethecommentsfromtheNSStoexplorethereasonsbehindthepatternsinthequantitativeresults.

29 Institutionsfindthecommentstobemoreeffectiveatengagingstaffindiscussionsaroundstudentfeedback.

30 Theuseofstudentandstafffocusgroups,especiallyatdepartmentalandcourselevel,allowmuchdeeperexplorationofstudents’experiencesoftheircourse.

59

5.2 Questions for reflection

Thefollowingarethe‘questionsforreflection’fromthroughoutthereport..Asstatedintheintroduction,theyareintendedtobeusedtoinspirediscussionandcriticalreflection.

Staff-student partnerships

1 ArestudentspresentonallthecommitteesandgroupswhereNSSresultsarediscussedinanydetail?

2 Arestudentrepresentativessufficientlysupportedtoallowthemtoplayanactiveroleinmeetings?

3 DoestheinstitutionandtheSUworktogetherontheNSS,inawaythatishonest,productive,sustainableandacknowledgesthechallengesthatexist?

4 DostudentshaveadirectroleintheexplorationandinterpretationoftheNSSresults,oristheirrolelimitedtothepromotionofthesurveyandattendingcommitteemeetingsasstudentrepresentatives?

5 IssupportprovidedtotheSUtoenablethemtoundertaketheirownexplorationoftheNSSdata?

6 Istherescopetorecruitstudents(includingpostgraduatestudents)aspaidresearchersanddataanalysts,ortoinvolvetheNSSinrelevantmodules,e.g.socialscienceresearchmethods?

7 ArestudentsawareofthevaluethatisplacedontheNSSresults,andtheimpactthatthoseresultshaveondecisionswithintheinstitution?

8 AretheoptionalquestionsintheNSS,thataskstudentsabouthowthefeedbackthattheyprovideisrespondedto(QuestionB.6),usedwithintheinstitution?Ifnot,doyouhaveotherwaysofgatheringthatinformation?

9 IfyouareunabletociteactionstakenindirectresponsetoNSSscores(becausedecision-makingismorecomplex)howelsecanyoumakecleartostudentsthatthesurveyresultsaretakenseriously?

10 Doesthetoneandmethodbywhichthemessagesarefedbacktostudentsembodytheinstitution’svisionofthestaff-student,andinstitution-student,relationship?

Institutional structures

11 Doestheinstitutionrespondtopoor-scoringareasintheNSSinasupportiveway?Areareasthatconsistentlyperformwellinthesurveyhighlighted?

12 HowmuchfreedomdoacademicshaveabouthowtheyrespondtoNSSresults?Aretheyconstrainedbyanoverlybureaucraticprocedurethatpreventsthemfromengagingandinnovating?

13 IstheNSSopenlydiscussedanddebated?Istherelevantresearchliteraturereadandapplied?

14 IstheNSSembeddedinyourplanningandreviewsystemsinawaythatpromptscriticalreflectionandinnovationbycourseteams?

15 AreacademicsencouragedtousestudentcommentsfromtheNSSalongsidethenumericalscoreswhenreviewingmodulesandcourses?

60

16 DoinstitutionalNSStargetssufficientlytakeintoaccountthelimitationsofthedata?

17 HastheNSScreatedagreaterinstitutionalfocusontheenhancementoflearningandteaching?Oristhemainimpactonmarketing,recruitmentandqualityassurance?

18 HasthenetimpactoftheNSSonstudentlearningbeenpositiveornegative?

19 HastheNSScontributedtoa‘studentasconsumer’approachwithintheinstitution,orhasithelpedtosupportagenuinedialogueandengagementwithstudents?

Analysis and exploration

20 DothebenefitsoftheNSSforqualityenhancementoutweighthedrawbacks?

21 IstheNSSagoodmeasureofeducationalquality?Whatothertypesofinformationprovideacomplementaryorsuperiormeasure?

22 DoestheNSScontributetoadamagingconsumerculture,orcanitbeharnessedtoimprovethepedagogicrelationship?

23 HowdoyouuseyourunderstandingofthelimitationsoftheNSStomakemoreeffectiveuseofitforenhancement?

24 IstheNSSusedasastand-alonemeasureofteachingquality,orittreatedasonetoolamongothers?

25 Whatinformationaboutlearningandteachingisrequiredtoinformenhancementactivities?HowdoestheNSSfitintothat?

26 IftheNSSjusttellsyouwheretolook,whatisthebestofwayofexploringfurther?Whatinformationwouldhelpyoudecidehowtorespond?

27 Whatspecificinformationareyoutryingtogainthroughtheprocessofmakingcomparisons?

28 Arebenchmarkingexercisesdesignedtoservemarketingandrecruitment,orqualityenhancement?

29 Doyoucomparedepartmentsandcourseswitheachotherwithintheuniversity,orwithsimilardepartmentsandcoursesaroundthesector?

30 DoesleaguetablepositionhaveanimpactontheresponsetotheNSS,regardlessofwhethertheleaguetablemethodologyisaccepted?

31 Whatistherightbalanceofformalandinformalmethodsforgatheringqualitativedataabouttheexperiencesofstudents?

32 Howshouldstudentcommentsbeusedinstaffmeetingsandfocusgroups,inawaythatmaximisesengagementanddiscussionbutissensitivetotheemotivenatureofstudents’comments?

61

5.3 Further reading

TherearenumberofstudiesandreviewsthatprovideusefulinformationaboutthestrengthsandlimitationsofNSSdata13.RelevantresearchproducedbytheHEAisdescribedinthenextsection.

AlanFielding,PeterDunleavyandMarkLangan(2010)InterpretingcontexttotheUK’sNationalStudent(Satisfaction)Surveyforsciencesubjects.Journal of Further and Higher Education. 34(3),347-368.

ThisisaninvestigationintothecomplexissuesthatcanarisewheninterpretingNSSdata.Anumberofimportantfindingsarecontainedinthearticle,suchastheabsenceofastrongcorrelationbetweentheexperienceoffeedbackandoverallsatisfaction,andtheimportantsubjectdifferencesinstudents’responsestotheNSSitems.

AbbiFlint,AnneOxley,PaulHelmandSallyBradley(2009)Preparingforsuccess:oneinstitution’saspirationalandstudentfocusedresponsetotheNationalStudentSurvey.Teaching in Higher Education.14(6),608-618.

ThisarticlediscussestheinvolvementofstudentsintheprocessofusingNSSdataforqualityenhancementpurposes.Variousactivitiesaredescribed,includinganeventtoallowacademicstohearstudentperspectivesindetail,andthepublicationofa‘Yousaid…wedid’documenttoinformstudentsofthechangesthathadresultedfromtheirfeedback.

LeeHarvey(2005)AhistoryandcritiqueofqualityevaluationintheUK. Quality Assurance in Education. 13(4),263-276.

Thispapermapsthedevelopmentofqualityevaluationproceduresandraisesquestionsaboutcurrentapproaches.HarveyconcludesthatqualityevaluationsintheUKhavebeenguidedasmuchbypoliticalpragmatismasconcernswiththequalityofteaching.

HEFCE(2011)National Student Survey: Findings and trends 2006 to 2010.Bristol:HEFCE.

ThisisthelatestannualreportontheNSSbyHEFCE.Itprovidesanoverviewofthe2010data,aswellaslookingattrendsinthedatafrom2006to2010aroundvariousdemographiccharacteristicsofthestudentpopulation.

MikeProsser(2005) Why we shouldn’t use student surveys of teaching as satisfaction ratings.York:HigherEducationAcademy.

Thisveryshortbutwidely-quotedpaperwaswrittenbyMikeProsserin2005whenhewasDirectorofResearchandEvaluationattheHEA.ItlaysouttheviewthatusingtheNSStoformulateleaguetablesmayhinderitsuseastoolforimprovingstudentlearningexperiences.

PaulRamsden,DeniseBatchelor,AlisonPeacock,PaulTempleandDavidWatson(2010)Enhancing and developing the National Student Survey: report to HEFCE.Bristol:HEFCE.

Thisreport,commissionedbyHEFCE,providedaninterimevaluationofthefunctionsandperformanceoftheNSS,inordertoarriveatrecommendationsaboutwhetherthesurveyshouldbeupdatedordeveloped.Thestudyproposednosubstantialchangestothesurvey,butrecommendedthatafullreviewbeundertakenin2015.

JohnRichardson(2005)Instrumentsforobtainingstudentfeedback:areviewoftheliterature.Assessment and Evaluation in Higher Education. 30(4),387-415.

13 Afullerannotatedbibliography,compiledbytheauthor,isavailableathttp://evidencenet.pbworks.com/w/page/28700535/NSS%20Resources

62

Thisisaveryusefulreviewoftheresearchliteratureconcerningthedifferentkindsofsurveytoolsthatcanbeusedtogatherinformationaboutstudents’learningexperiences.

JohnRichardson,JohnSlaterandJaneWilson(2007)TheNationalStudentSurvey:development,findingsandimplications.Studies in Higher Education. 32(5),557-580.

ThisarticledescribesthehistoryanddevelopmentoftheNSS,focusingonthemechanismsandfindingsofthetwopilotsurveysthattookplacein2003and2004.

JamesWilliamsandGillCappucini-Ansfield(2007)Fitnessforpurpose:Nationalandinstitutionalapproachestopublicisingthestudentvoice. Quality in Higher Education. 13(2):159-172.

Thisarticlecomparesthestrengthsandweaknessesofnationalsurveys(whichhavelargesamplesandprovidecomparability)andinstitutionalsurveys(whichcanbetailor-madeforparticularcontexts).Itfocusesonthevalueofstudentengagementwithsurveys.

RuthWilliamsandJohnBrennan(2003)Collecting and using student feedback on quality and standards of learning and teaching in HE.Bristol:HEFCE.

ThisisareportcommissionedbyHEFCEinorderto:i)toidentifygoodpracticeincollectingfeedbackfromstudents,forqualityenhancement;andii)tomakerecommendationsaboutthedesignandimplementationofanationalsurveyofstudents.ThisreportplayedanimportantroleinthedevelopmentoftheNSS.

KeithiaWilson,AlfLizzioandPaulRamsden(1997)Thedevelopment,validationandapplicationoftheCourseExperienceQuestionnaire. Studies in Higher Education. 22(1),33-53.

ThispaperexaminesthehistoryoftheCourseExperienceQuestionnaire(CEQ),andtheempiricaldataconcerningitsusefulnessasaperformanceindicatorofteachingquality.AsthedesignoftheNSSwasbasedontheCEQ,thispaperprovidesusefulinformationaboutthevalidityoftheNSS. 

MantzYorke(2009)Studentexperiencesurveys:somemethodologicalconsiderationsandanempiricalinvestigation.Assessment & Evaluation in Higher Education. 34(6),721-739.

Thisarticlelooksatanumberofissuesandcontroversiesaroundthedesignandadministrationofsector-widestudentsurveys,includingtheNSS.

63

5.4 Additional HEA resources

TheHigherEducationAcademysupportsinstitutionsanddisciplinecommunitiestousestudentsurveydatatoenhancethestudentlearningexperience.FormoreaboutourworkontheNationalStudentSurvey,andtodownloadourreports,pleasevisit:http://www.heacademy.ac.uk/nss.

Data reports

TheHEAhasproducedarangeofreportsusingthe2011NSSresults.

• 28disciplinereports,containingdetaileddataandcomparisonsfor67subjects.

• Anational-levelreport,whichincludesincludingcomparisonsforage,gender,modeofstudy,andarangeofotherstudentandinstitutioncharacteristics.

Research

TheHEAhasproducedanumberofkeypiecesofresearchrelatingtotheNSS:

Dimensions of quality (2010)

ProducedbyGrahamGibbs,thisreportsetsouttoidentifythosefactorsthatgiveareliableindicationofthequalityofstudentlearning.Itsfocusisbroaderthanjusttheuseofstudentsurveydata,butitprovidesausefuloverviewofdifferentmechanismsofevaluatingeducationalquality.

Availablefrom:http://www.heacademy.ac.uk/assets/documents/evidence_informed_practice/Dimensions_of_Quality.pdf.

The National Student Survey three years on: What have we learned? (2009)

ThisreportbyPaulaSurridgesummarisessomekeypiecesofresearchtogiveanoverviewoffindingsrelatingtotheNSS.Italsogivesrecommendationsforfuturework.ItisaveryusefulguidetoNSSdata,especiallyregardingtheimportantquestionofwhatitcanandcannottellus.

Availablefrom:http://www.heacademy.ac.uk/assets/documents/research/surveys/nss/NSS_three_years_on_surridge_02.06.09.pdf.

National Student Survey of Teaching in UK Universities: Dimensionality, multilevel structure and differentiation at the level of university and discipline: preliminary results (2008)

Thisreport,byHerbMarshandJacquelineCheng,isatechnicalinvestigationofanumberofissues,focusinginparticularontherelativeeffectsonNSSscoresofvariousfactorssuchasinstitutionanddiscipline.ItisarichsourceofinformationthatcanhelptoilluminaterawNSSdata.

Availablefrom:http://www.heacademy.ac.uk/assets/documents/research/surveys/nss/NSS_herb_marsh-28.08.08.pdf.

64

Postgraduate surveys

InadditiontosupportingthesectortouseNSSdatafortheenhancementoflearningandteaching,theHEAhasalsodevelopeditsownnationalsurveys,lookingatthepostgraduatestudentexperience.

Postgraduate Taught Experience Survey

PTEShasbeenrunningsince2009,andin2012about54,640studentsfrom83institutionscompletedthesurvey.Thesurveyasksstudentsaboutawiderangeofelementsoftheirlearningexperience,includingfeedback,teachingandskillsdevelopment.Italsoasksaboutthedepthandsophisticationofthelearningtheyhaveengagedin.

Formoreinformationvisit:http://www.heacademy.ac.uk/ptes.

Postgraduate Research Experience Survey

PRESisthesistersurveyofPTESandisaimedatpostgraduateresearchstudents.Itrunseverytwoyears,andin2011over31,000studentsfrom102institutionscompletedthesurvey.Thesurveywillnextrunin2013.

Formoreinformationpleasevisit:http://www.heacademy.ac.uk/pres.

Consultancy and change programmes

TheHEArunsregularchangeprogrammesfordepartmentsandfacultieswishingtoexploretheirNSSresults.Moreinformationcanbefoundhere:http://www.heacademy.ac.uk/change.

TheHEAisalsocurrentlydevelopinganinstitutionalconsultancyservice,whichwillprovideseniormanagerswithadvice,tailoredanalysisandsupporttohelpthemusesurveydatatostrategicallyaddressissuesinlearningandteaching.Ifyouareinterestedinthisservicethenpleaseemail:nss@heacademy.ac.uk.

65

Appendix A: Information about the NSS

TheNSSisasurveyoffinal-yearstudentsonundergraduateprogrammes.ItiscompulsoryforpubliclyfundedHEprovidersinEngland,WalesandNorthernIreland,andsomeScottishinstitutionstakepartonavoluntarybasis14.IpsosMORIadministerthesurveyonbehalfofHEFCE,andcontactallsuitablestudentsusingavarietyofmethods(includingemailandtelephone).Thesurveywasintroducedin2005,andin2012154HEIsand106FECstookpart,and287,000studentsresponded–anoverallresponserateof67%.

NSSdataarecurrentlyavailableprimarilyfromtheUnistatswebsite(http://unistats.direct.gov.uk),whichallowsvisitorstocompareoverallsatisfactionresultsatcourseandinstitutionallevel,aswellasdownloadspreadsheetswithmorecomprehensiveinformation.InadditionHEFCEreleasesheadlinefigures,aswellasannualreportsprovidingnational-levelanalysis.FromSeptember2012,course-levelNSSdatahavebeenincorporatedintoKeyInformationSets,which,aswellasbeingavailableonUnistats,willbeembeddedoninstitutionalwebsites.

Forreasonsofreliabilityandconfidentiality,thethresholdforpublicreportabilityoftheresultsisaresponserateof23responses,whichmustalsorepresentatleast50%oftheeligiblestudents.Wheretherearelessthan23responses,responsesfrommorethanoneyear,orfromacrossdifferentcourses,canbeaggregatedtoproducepubliclyreportabledata.

Inadditiontothepublicavailabilityofthedata,institutionsreceivetheirowndataatamoredetailedsubjectlevel.Thereportabilitythresholdforthedatathatinstitutionsreceiveistenresponses,ratherthan23.DataattheindividualstudentlevelarealsoavailableforresearchersonapplicationtoHEFCE.

TheNSSisbasedtoasignificantextentontheCourseExperienceQuestionnaire(CEQ),whichhasbeeninuseinAustraliasince1993.TherehasbeenasignificantamountofresearchontheCEQ,andamorelimitedamountontheNSS,andthisresearchindicatesthatthetwosurveysarebothreliable–theyyieldconsistentandrepeatabledata–andvalid–theymeasurewhattheypurporttomeasure.

TheNSSasksparticipantstoratetheirlevelofagreementwith23positivestatements,onafive-pointscale(inadditionto‘notapplicable’):definitelydisagree;mostlydisagree;neitheragreenordisagree;mostlyagree;definitelyagree.Thestatementsaregroupedintosixareas,or‘scales’,plustheoverallstatementandastatementabouttheStudents’Union:qualityofteachingandlearning;assessmentandfeedback;academicsupport;organisationandmanagement;learningresources;personaldevelopment.

Aswellasaskingparticipantstoratetheiragreementwith23statements,thesurveyalsoinvitesthemtoaddfree-textcommentsaboutparticularpositiveornegativeaspectsoftheirexperience.Institutionscanchoosetoutiliseabankofoptionalstatementsinadditiontothe23corestatements,whicharenotpubliclyreported.

14 15Scottishinstitutionstookpartin2012.

66

Appendix B: Core NSS items15

The teaching on my course

1 Staffaregoodatexplainingthings2 Staffhavemadethesubjectinteresting3 Staffareenthusiasticaboutwhattheyareteaching4 Thecourseisintellectuallystimulating Assessment and feedback

5 Thecriteriausedinmarkinghavebeenclearinadvance6 Assessmentarrangementsandmarkinghavebeenfair7 Feedbackonmyworkhasbeenprompt8 Ihavereceiveddetailedcommentsonmywork9 FeedbackonmyworkhashelpedmeclarifythingsIdidnotunderstand Academic support

10 Ihavereceivedsufficientadviceandsupportwithmystudies11 IhavebeenabletocontactstaffwhenIneededto12 GoodadvicewasavailablewhenIneededtomakestudychoices Organisation and management

13 Thetimetableworksefficientlyasfarasmyactivitiesareconcerned14 Anychangesinthecourseorteachinghavebeencommunicatedeffectively15 Thecourseiswellorganisedandisrunningsmoothly

Learning resources

16 Thelibraryresourcesandservicesaregoodenoughformyneeds17 IhavebeenabletoaccessgeneralITresourceswhenIneededto18 Ihavebeenabletoaccessspecialisedequipment,facilities,orroomswhenIneededto

Personal development

19 Thecoursehashelpedmetopresentmyselfwithconfidence20 Mycommunicationskillshaveimproved21 Asaresultofthecourse,Ifeelconfidentintacklingunfamiliarproblems

Overall satisfaction

22 Overall,Iamsatisfiedwiththequalityofthecourse

Students’ Union (Association or Guild)

23 IamsatisfiedwiththeStudents’Union(AssociationorGuild)atmyinstitution

15 AnelectroniccopyoftheNSSquestionnaireisavailablehere:http://www.thestudentsurvey.com/content/nss2012_questionnaire_english.pdf[accessed23October2012].AfulllistoftheNSSquestions,includingtheoptionalquestions(butpriortotheintroductionofQ23),isavailablehere:http://www.ipsos-mori.com/Assets/Docs/Publications/nss-questionnaire.pdf[accessed23October2012].

67

Contact us

The Higher Education Academy Innovation Way York Science Park Heslington York YO10 5BR

+44 (0)1904 717500 enquiries@heacademy.ac.uk

ISBN number: 978-1-907207-65-5

© The Higher Education Academy, 2012

The Higher Education Academy (HEA) is a national body for learning and teaching in higher education. We work with universities and other higher education providers to help bring about change in learning and teaching. We do this to improve the experience that students have while they are studying, and to support and develop those who teach them.

Our activities focus on rewarding and recognising excellence in teaching, bringing together people and resources to research and share best practice, and by helping to influence, shape and implement policy - locally, nationally, and internationally.

www.heacademy.ac.uk | Twitter @HEAcademy

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any storage and retrieval system without the written permission of the Editor. Such permission will normally be granted for educational purposes provided that due acknowledgement is given.

To request copies of this report in large print or in a different format, please contact the communications office at the Higher Education Academy: 01904 717500 or pressoffice@heacademy.ac.uk