reflection report (io6) and assessment of pilot peer ... · assessment of pilot peer reviews and...
TRANSCRIPT
Universität für Bodenkultur Wien University of Natural Resources and Life Sciences, Vienna
ReflectionReport(IO6)andAssessmentofPilotPeerReviews(O7)Finalreport
MariaGutknecht-Gmeiner,Susanna-MariaHenkel,SophieKroiss,ChristinaPaulus
EuropeanPeerReviewAssociation(EPRA),UniversitätfürBodenkultur
Vienna,May2018
Project"TransnationalPeerReviewforqualityassuranceinValidationofNonFormalandInformalLearning(VNFIL)Extended"(Projectno.2015-1-NL01-KA204-009004)
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 2
AssessmentofPilotPeerReviews(IO7)andReflectionReport(IO6)
MariaGutknecht-GmeinerVienna,May2018Imprint:EuropeanPeerReviewAssociationDr.-Josef-Resch-Pl.14/3,1170WienTel.:+436642365980E-Mail:[email protected]://www.Peer-review-network.euhttp://www.Peer-review-vnfil.eu
ThisprojecthasbeenfundedwithsupportfromtheEuropeanCommission.Thispublicationreflectstheviewsonlyoftheauthors,andtheCommissioncannotbeheldresponsibleforanyusethatmaybemadeofthein-formationcontainedtherein.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 3
CONTENTS
1. BackgroundandaimsoftheprojectPeerReviewVNFILExtended.................................................61.1 EuropeanPeerReview........................................................................................................61.2 TheprojectPeerReviewVNFILExtended...........................................................................6
2. Aims,topicsandmethodsofthequalitativestudyandassessment...............................................72.1 Formativeandsummativeaimsandapproach..................................................................72.2 Researchquestions.............................................................................................................82.3 Designandmethods...........................................................................................................82.4 Availabledocumentationandresponserates....................................................................9
3. PeerReviewPilots.........................................................................................................................113.1 Organisationofpilots.......................................................................................................113.2 ParticipationinPeerReviewpilots...................................................................................133.3 QualityAreas....................................................................................................................143.4 Peers.................................................................................................................................153.5 PeerTrainingandotherformsofsupport........................................................................19
3.5.1 Trainingandsupportoffered..................................................................................................193.5.2 Participationintrainingsandotherformsofsupport.............................................................203.5.3 Feedbackontrainingsandsupportduringpreparatoryphase...............................................223.5.4 Levelof“preparedness”ofVNFILprovidersandPeers...........................................................23
3.6 PreparationofthePeerReview(Phase1)........................................................................233.6.1 PreparationoftheVNFILProviders.........................................................................................233.6.2 Self-Report..............................................................................................................................233.6.3 PreparationinthePeerTeams................................................................................................24
3.7 PeerVisit(Phase2)...........................................................................................................253.7.1 PeerVisitAgendas...................................................................................................................253.7.2 ExperiencesduringPeerVisits................................................................................................27
3.8 PeerReviewReport(Phase3)..........................................................................................293.8.1 Feedbacksessionandusefulnessoffeedback........................................................................293.8.2 Timelinessofreports...............................................................................................................293.8.3 Writingthe“PeerReviewReport”..........................................................................................293.8.4 PeerConsulting.......................................................................................................................29
3.9 Puttingplansintoaction(Phase4)...................................................................................303.9.1 Communicationofresults.......................................................................................................30
3.10 Cooperation,rolesandtransnationalaspects..................................................................313.10.1 QualityofcooperationduringPeerReviews.........................................................................313.10.2 BenefitsofmultipleparticipationofPeersandfacilitators...................................................313.10.3 LessonslearnedfromnationalandinternationalcooperationduringPeerReview.............313.10.4 Transnationalaspects:TheimportanceofinternationalPeers............................................32
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 4
4. EffectsofthePeerReview.............................................................................................................334.1.1 Institutional„returnoninvestment“ofthePeerReview.......................................................334.1.2 RelevantchangesintermsofinstitutionaldevelopmentsincethePeerReview...................334.1.3 GoodpracticesinVNFILduringPeerReviewtransferredtootherinstitutions/countries.....344.1.4 LessonslearnedduringthePeerReviewintermsofinstitutionalQAandQM......................34
5. GeneralassessmentofPeerReview..............................................................................................355.1 ApplicabilityandaddedvalueofPeerReviewforVNFIL..................................................355.2 Potentialof(transnational)PeerReviewforthedevelopmentofVNFILandquality
assuranceinVNFIL............................................................................................................355.3 FurtherdevelopmentofinstitutionalqualityassurancethroughPeerReview................355.4 Challengesforimplementing(transnational)PeerReviewinVNFIL................................36
6. FeedbacktoManualandToolboxandrecommendationsforfinalisation....................................376.1 Manual..............................................................................................................................376.2 QualityAreas....................................................................................................................376.3 Toolbox.............................................................................................................................37
7. Documentsand(re)sources...........................................................................................................397.1 Basicdocuments...............................................................................................................397.2 Documentationofpilotphaseandmonitoringdata........................................................397.3 Reportsandpublications..................................................................................................397.4 Listofinterviewpartners..................................................................................................397.5 Participantsreflectionworkshop......................................................................................407.6 Additionaltables...............................................................................................................41
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 5
Tableofexhibits
Figure1: PhasesoftheEuropeanPeerReview...................................................................................7Table2: OverviewofdocumentsdeliveredbyVNFILproviders.........................................................9Table3: Surveyresponses................................................................................................................10Table4: ParticipatingVNFILprovidersandPeerVisitdates.............................................................13Table5: NumberofdeploymentsperPeer......................................................................................13Figure6: NumberofPeerReviewswith…Peers...............................................................................14Table7: QualityareaschosenbyVNFILproviders............................................................................15Figure8: CountriesofPeers...............................................................................................................16Figure9: FunctionsofPeersduringreviewofowninstitution..........................................................16Figure10: InstitutionalbackgroundsofactivePeers...........................................................................17Figure11:PreviousQualityAssuranceTrainingofPeers....................................................................17Table12: ReviewskillsofPeers(self-assessment).............................................................................18Figure13:Participationin(typesof)PeerTrainings............................................................................20Figure14:NºofPeerTrainingsperPeer.............................................................................................21Figure15:OtherformsofsupportforPeers(multipleanswerspossible)..........................................21Table16: FeedbacktoEuropeanPeerTraining..................................................................................22Table17: Levelof“preparedness”ofthePeers.................................................................................23Table18: PeerVisitAgendas:Numberofsessions,numberofinterviews
withdifferentinterviewgroups..........................................................................................25Table19: RepresentationofcandidatesinPeerVisit.........................................................................26Table20: QualityofcooperationinPeerTeam..................................................................................27Table21: QualityofcooperationbetweenProviderandPeers.........................................................31Table22: PeersperPeerReview........................................................................................................41Table23: Peers:distributionbysex...................................................................................................41Table24: Peers:distributionbycountry............................................................................................41Table25: InstitutionalbackgroundofPeers.......................................................................................42Table26: TypesofPeerTrainingcompletedbyPeers.......................................................................42Table27: NºofPeerTrainingsperPeer.............................................................................................42Table28: PreviousQualityAssuranceTrainingofPeers....................................................................43Table29: Review/evaluationskillsofPeers(self-assessment)...........................................................43Table30: ParticipationofPeersinPeerReviewofowninstitution...................................................43Table31: Howwasthecooperationwithinyourinstitution
inthepreparationandconductofthePeerReview?.........................................................44Table32: WasthetimeforthePeerVisitsufficient?.........................................................................44
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 6
1. BackgroundandaimsoftheprojectPeerReviewVNFILExtended
1.1 EuropeanPeerReview
PeerReview is a formofexternalevaluationwith theaimof supporting the reviewededucationalinstitution in its quality assurance and quality development efforts. An external group of experts,calledPeers,isinvitedtoassessthequalityofdifferentfieldsoftheinstitution,suchasthequalityofeducationandtrainingprovisionofindividualdepartmentsoroftheentireorganisation.Duringtheevaluationprocess,thePeersvisitthereviewedinstitution.Peersareexternalbutworkinasimilarenvironmentandhavespecificprofessionalexpertiseandknowledgeoftheevaluatedsubject.Theyareindependentand"personsofequalstanding"withthepersonswhoseperformanceisbeingre-viewed.PeerReviewhasinthepast10yearsbeenadaptedfromhighereducationtovocationaledu-cationandtraininginaseriesofEuropeanprojectsthatwerecloselylinkedtoEQAVET.
TheEuropeanPeerReviewprocedureprovidesaqualityassuredprocedureandcommonstandardforconductingPeerReviewsacrossEurope.TheprocedureisdocumentedintheEuropeanPeerRe-viewManualandToolbox.Itwasoriginallydevelopedintheareaofvocationaleducationandtrain-ingintheyears2000(PeerReviewininitialVET,2004-2007;PeerReviewExtended,2007;PeerRe-viewExtendedII,2007-2009).After2009,theEuropeanPeerReviewwasintroducedasanexternalinstrumentforqualityassuranceinthevocationaleducationandtrainingsystemsinvariousEurope-ancountries,amongwhichareFinland,Italy,Hungary,CataloniaandAustria.
OnaEuropeanlevel,transferofPeerReviewwasbuttressedbysubsequentcallsforprojectsintheLifelongLearningProgramme(priority for transferof innovationprojects)whichresulted in furtherexperimentationofPeerReview invocationalguidanceandcounselling (EuroPeerGuid,2010-2012)and,from2014onwards,alsointheareaofrecognitionofpriorlearninginPortugal,France,andtheNetherlands(EuroPeerguid-RVC,2014-2015).1
1.2 TheprojectPeerReviewVNFILExtended
The'EuropeanPeerReviewVNFILExtended'projecttakesupfromthesepreviousprojects.ItseekstointroducePeerReviewasaninstrumentofqualityassuranceandqualitydevelopmenttothevali-dationofnon-formalandinformal learning(VNFIL). Inparticular, itbuildsupontheresultsofEuro-Peerguid-RVCwhich itaims toextend– tonewcountries (Austria, Lithuania,Slovakia), to transna-tional useof PeerReviewand to a sustainable integrationof PeerReview into thenational – andpotentiallyalsoEuropean–qualitystrategiesforVNFIL.
Adecisiveelementof theprojectwasto test theapplicabilityof theEuropeanPeerReviewproce-durein11pilotPeerReviewsin6oftheparticipatingcountries,adoptingatransnationalapproach,i.e.withhalf thePeer Teamcoming fromanother country (O5). The latterwas intended to fostermutuallearningbetweenVNFILprovidersacrossborders,enhancingtransferofinnovationinVNFILand enhancingmutual trust in the quality of VNFIL provision in Europe. The pilots took place be-tweenJune2016andMay2017andwereheadedoffbyajointEuropeanPeerTraining(June2016).
Acomprehensiveassessmentandin-depthqualitativeinvestigationofthepilotphasewascrucialfordeterminingtheusefulnessofPeerReviewforthequalitydevelopmentofVNFILprovision.Itprovid-edabasisforthedevelopmentorfinalisationofthemajorityofprojectoutcomes:§ thefinalisationoftheManual(O2),theToolbox(O3),andtheQualityAreas(O3)§ thedevelopmentof thebusinessmodel (O7), thenational strategies for PeerReview inVNFIL
(O8)aswellasthePeerReviewReader(O9)andtheEuropeanpolicypaper(O10).
1Informationonthe„history“oftheEuropeanPeerReviewandpastprojectscanbefoundonthewebsiteoftheEuropeanPeerReviewAssociationwww.Peer-review-network.eu.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 7
2. Aims,topicsandmethodsofthequalitativestudyandassessment
2.1 Formativeandsummativeaimsandapproach
Theoverallaimoftheassessmentandqualitativestudyofthepilotphasewastoappraisethefeasi-bilityandsuitabilityoftheuseoftheEuropeanPeerReviewprocedureinVNFILwiththeprospectofapossibleexpansiontootherVNFILinstitutions.ThisalsoincludedaninvestigationoftheusefulnessandaddedvalueofPeerReviewforthePeersandtheVFNILinstitutionsinvolved.Assessingthesuc-cess of the transfer of PeerReview toVNFIL thuswas the summativepartof the investigation. Inaddition,theassessmentandqualitativestudywantedtocontributetofurtherimprovementoftheorganisation,preparationandconductofEuropeanPeerReviewsintermsofaformativeassessment.
Atwo-stepapproachwastakentoproducevalidanddependableresults:§ Anin-depthexaminationoftheelevenpilotPeerReviewsascertainedthequalityoftheprocess
(includingtrainingandsupportbythecoordinatingbody)andthelevelofimplementationfidelityintermsofadherencetotheEuropeanPeerReviewprocedureaslaiddownintheManualandtheToolbox(O7).
§ Basedonthis,thequalitativestudy(i.e.thereflectionreport)wascarriedout(O6).
Sincethetwostepsareinseparablyinterlinked–thelatterpartcannotbedoneinaseriousmannerwithoutthepriorprocessassessment–thisreport isbasedonacoordinateddatacollectioneffortbetweenthetworesponsibleinstitutionsandcomprisestheoutcomesofbothO6andO7.Thefideli-ty assessment also constitutes integral part of the Peer Review label award (O7). It followed thephasesofthePeerReviewasdescribedintheManual.
Figure1: PhasesoftheEuropeanPeerReview
Source: EuropeanPeerReviewManualforVNFIL
Phase1:Prepara'on(min.3months)• Ge4ngstarted• Invi'ngPeers• Self-Evalua'onsandSelf-Report• PreparingthePeerVisit
Phase2:PeerVisit(1.5-2days)• Collec'ngdata• Analysingdata• OralFeedback
Phase3:Peerreport(4weeks)• DraOReport• CommentsoftheVNFILprovider• Finalreport
Phase4:Pu4ngplansintoac'on(6-12months)• Formula'ngtargets• Clarifyingresources• Ac'onPlanandImplementa'on• PlanningthenextReview
NextPeerReview
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 8
2.2 Researchquestions
Part1:Processevaluationandassessmentofimplementationfidelity§ How(well)werePeersandVNFILproviderspreparedforthepilotPeerReviews?Whatwerethe
contributions of the European Peer Training?What other support did Peers and providers re-ceive?Howcouldtheprovisionoftrainingandsupportbeimproved?
§ Howwere thePeerReviewsconducted?Were thequality standardsof theEuropeanPeerRe-viewprocedure(Manual,Toolbox)adheredto?Howcantheprocessbeimproved?
Part2:Qualitativestudy§ IsthePeerReviewmethodologysuitableforVNFIL?Isthetrainingprogrammeappropriate?Are
thequalityareasandindicatorssuitableforVNFILproviders?HowcanthePeerReviewmethod-ology(Manual,Toolbox)stillbeimprovedandfine-tunedtoVNFIL?
§ WhataretheprosandconsofinternationalPeerReview?HowdoyouassessthecontributionofinternationalPeers?HowdidtheprocessworkwithtwointernationalPeers?WhatwerebenefitsordrawbacksofmultipleparticipationofPeersandfacilitatorsandmutualPeerReviews?
§ WhatkindofimpactdidthePeerReviewshaveontheinstitutionsandonthePeers?
§ Whatistheaddedvalueof(transnational)PeerReviewsforinstitutionalQAandQMandistheintegrationofPeerReviewapossiblefurtherdevelopmentofinstitutionalqualityassurance?
§ What are the possibilities and challenges in further implementation of transnational Peer Re-viewsforVNFIL?Istherepotentialforimplementing(transnational)PeerReviewasaformativeexternalevaluationforVNFILonnationaland/orEuropeanlevel?
2.3 Designandmethods
Thecombinedassessmentandstudyfollowedamixed-methoddesignwithatriangulationofdiffer-entstakeholders,sourcesandmethodsincludingbothquantitativeandqualitativeelements.Awiderangeofdataandsourceswereusedwith
§ “hard data” from the documentation of the Peer Reviews (documents, monitoring data) andonlinesurveysgatheringinformationandfeedbackfromparticipatingPeersandProviders(O7)
§ complementedbyaworkshopwithpartnersandqualitative interviewsaswell asaqualitativeanalysisofsurveyresponses(openquestions)andselectedmonitoringdocuments,inparticular,themeta-evaluationsofPeers(O6).
1. MonitoringdataonthepilotPeerReviews
§ PeerReviewdatabase:overviewofpilotPeerReviews
§ PeerdatabasecontainingallinformationfromPeerapplications
§ PeerReviewdocuments(Toolbox)§ Self-Reports§ PeerReviewAgendas§ Meta-evaluationsofPeers§ PeerReviewReports§ OtherdocumentationofPeerReview,ifexistent:presentations,documentationoffeed-
backsession,interviewandobservationguidelines,interviewprotocols,photos
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 9
2. OnlinequestionnairesforparticipantsinpilotPeerReviews
InSeptember2017,anonlinesurvey(usingtheonlinetoolLimeSurvey)wasconducted,encompass-ingquantitativeandqualitativequestions.Theyincludedseparatequestionnairesfor
§ Peersand
§ VNFILproviders(PeerReviewfacilitators/coordinators,managersofreviewedcentres).
3. AworkshopwiththeprojectpartnersintheformofaWorldCaféaspartoftheprojectmeetinginMay/June2017
4. Qualitativeinterviewswith7projectpartnersbasedonaninterviewguide
Cf.Listofinterviewpartnersinannex
2.4 Availabledocumentationandresponserates
Databasesandpilotdocumentation(monitoring)
Thepurposeof thedocumentationof thePeerReviewsand theestablishmentofapilotdatabasewastosupporttheplanning,monitoringandsteeringofthepilotPeerReviews.
Table2: OverviewofdocumentsdeliveredbyVNFILproviders
Code
Initialinform
ation
sheet
Selfrepo
rt
Agen
dafo
rPeerV
isit
Otherdocum
entatio
nPe
erVisit
Metaevalua
tion
ofPeers
PeerReviewRep
ort
PeerApp
lications
16_01_NL ✓ ✓ ✓ minutesof3interviews ✓ ✓ ✓16_02_AT ✓ ✓ ✓ ample ✓ ✓ ✓17_03_LT ✓ ✓ ✓ attendancelist ✓ ✓ ✓17_04_FR ✓ ✓ ✓ ample ✓ ✓ ✓17_05_PT ✓ ✓ ✓ minutesof11interviews,
finalpresentation ✓ ✓ ✓
17_06_AT ✓ ✓ ✓ ample ✓ ✓ ✓17_07_AT ✓ ✓ ✓ questionsmatrix,findingsof
PeerTeam ✓ ✓ ✓
17_08_PT ✓ ✓ ✓ minutesof6interviews,findings ✓ ✓ ✓
17_09_AT ✓ ✓ ✓ ample ✓ ✓ ✓17_10_NL ✓ ✓ ✓ minutesof6interviews,
QAAssessment ✓ ✓ ✓
17_11_SK inSelf-Report ✓ ✓ attendancelist ✓ ✓ ✓except1
Source: PilotDatabase;documentationofpilots
Allmaindocumentswereprovided inEnglish. The coordinatingbodyalso collectedPeer contractsandphotosdocumentingthePeerReviews.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 10
Onlinequestionnaires
ThequestionnaireswereonlinefromSeptember1sttoOctober16th,2017(providers)andfromSep-tember4thtoOctober31st,2017(Peers).Responseratesforthequestionnaireswere100%forVNFILprovidersand89,3%forPeers. (2Peersfilled inonequestionnairetogether,whichwascountedastworesponsesintheresponseratesaswellasintheoverallanalysis.)
Table3: Surveyresponses
CountryVNFILproviders Peers
totalnumber responses total
number responses
Austria(AT) 4 4 9 9
France(FR) 1 1 1 1
Lithuania(LT) 1 1 5 4
Netherlands(NL) 2 2 6 4
Portugal(PT) 2 2 4 4
Slovakia(SK) 1 1 3 3
Total 11 11 28 25
Source:OnlinesurveyofVNFILProvidersandPeers
Up to twodirect reminderswere sent to respondents via LimeSurveyduring theonline period (incaseofPeersupto3remindersforsome).Inaddition,emailsweresentdirectly,or(incaseofPeers)viamediatingprojectpartners.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 11
3. PeerReviewPilots
3.1 Organisationofpilots
Organisationalstructure
Organisationina largernetworkrequiresanactivecoordinatingbodyasaskedfor intheEuropeanPeerReviewManual.Thetaskofthecoordinatingbodyisto§ managethewholeprocessofconductingthePeerReviews,§ ensureacoherentapproach,§ providesupportandguidancetoProvidersandPeers,§ checkimplementationfidelity§ conductconstantmonitoring§ andsupportevaluation.
In theproject, thepartof the coordinatingbodywasassumedby thepartner Libereaux (aidedbyprojectcoordinatorErikKaemingk).LibereauxwasalsothepartnerresponsibleforIntellectualOut-come5.Generalmanagementandsupportremainedwiththeprojectcoordinator.EPRAwasrespon-sibleforpreparingtheassessmentofthepilots,actedas“sparringpartner”forthecoordinatingbodyandassumedanadvisoryroleprovidinggeneralorientationandexpertise.
Anonlineplatform(Shareboard,www.share-board.nl)wasinstalledfortheprojectandmanagedbyErikKaeminkC.V.AlthoughindividualspacesweresetupforallPeerReviews,theplatformwasnotwidelyused in thepilotphase since findingmaterialaswellasuploading/downloadingdocumentswasnotaseasyasthepartnershadexpected.Documentswerethuspassedontothecoordinatormainlyviaemail.EPRAprovidedaDropbox folder forsharing informationbetweenthe threepart-nerssteeringtheprocess,whichwasthenalsousedtostorecomprehensiveinformationonthepilotphaseandmakeitaccessibleforanalysis.
Process
ThePeerReviewswerescheduledforthesecondprojectphase,June2016–May2017andconduct-edbetweenNovember2016andMarch2016.
RegistrationforthePeerTrainingwasavailablefromendofJanuary2016,theonlinePeerapplicationtoolwasopenstartingmid-May2016.Thecoordinatingbodydevisedaplan(“Peerpuzzle”)formatch-ingPeersand institutionsforthePeerReviews.ThechallengewastoorganisethePeerReviews inawaythatallteamsincludenationalandatleast2internationalPeers.ThecoordinatingbodyalsoaskpartnerstostartpreparingthepilotphasebeforethePeerTrainingsothattheindividualplanscouldberevised,ifnecessary,andalignedwithanoverallactionplanuponduringthetraining.
ThetoolboxwasreadybeforethePeerTraining.However,anupdatedversionoftheQualityAreaswitharevisedstructureofcriteriawasonlypresentedduringthePeerTraining.SincepartnershadalreadystartedthepreparationoftheinitialphaseofthePeerReview(decidinguponQualityAreas,settingupaninternalstructure,requirementsforPeers),thisrequiredsomechanges.Aftersomeinitialconfusiontheadaptationtothe“new”QualityAreasoveralldidnotposeaproblemforpartners.
AcommonEuropeanPeerTrainingprecededthepilotPeerReviews:ItwasheldbetweenJune13-17,2016, inVienna, in theNetherland, anadditionalnational training followed (cf. III.6) inwhichalsonationalPeersandotherstakeholdersparticipated.ThetrainingwasalsousedtopreparethepilotphaseandcomeupwithamasterplanforthePeerReviewsaswellasmatchingPeersandinstitu-tions.ContrarytopreviouspilotsadduetothefundingrequirementsofErasmus+,atleasttwoin-ternationalPeersparticipated ineveryPeerReviewandthePeerpoolwas largelymadeupofper-sonneloftheprojectpartnersandonlyafewexternalPeers.ThisresultedinahighmutualityofPeerReviewswithparticipatingprovidersexchangingPeers.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 12
Amemberof thecoordinatingbodyalsoparticipated inallPeerReviews (MarloesSmit fromLibe-reauxorErikKaemingk)ensuringhighconsistencyofapproach,direct support forPeerTeamsandProviders andongoing secondorder learning concerning thePeerReviewprocess during thepilotphase.
Documentationandmonitoringofthepilotphasewascarriedoutbythecoordinatingbodyfromthestartandaccompaniedthewholepilotphase. Itallowedforaconstantmonitoringandsteeringofthepilotsandprovidedaverygooddatabasefortheassessmentandthestudy.
AninternalmentoringschemewassetupaccordingtoplanduringthePeerTraining.Itwas,howev-er, not usedmuchduring the initial stages of the PeerReviews sincepartners had ample supportthroughthecoordinatingbody(cf.Mentoringreport).
Projectteam“PeerReviewVNFILExtended”,June2016,Vienna(duringEuropeanPeerTraining)
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 13
3.2 ParticipationinPeerReviewpilots
ThefollowingoverviewshowstheparticipatinginstitutionsandthetimetableforthePeerVisits:
Table4: ParticipatingVNFILprovidersandPeerVisitdates
Code VNFILprovider Date
16_01_NL LibereauxBV,TheNetherlands Nov.24-25,2016
16_02_AT Weiterbildungsakademie(wba),Austria Dec.6-7,2016
17_03_LT VilniusVocationalTrainingCentreforServiceBusi-nessSpecialists,Lithuania Jan.18-19,2017
17_04_FR CentreInterinstitutionneldeBilandeCompétences(CIBC),BourgogneSud,France Feb.13-14,2017
17_05_PT CITEFORMA,Portugal Feb.2-3,2017
17_06_AT UniversityofNaturalResourcesandLifeSciences(BOKU),Unitoflifelonglearning,Austria March22-23,2017
17_07_AT FrauenstiftungSteyr,Austria Apr.4-5,2017
17_08_PT ISLASantarém,Portugal March9-10,2017
17_09_AT AKSalzburg/BFI,Austria May8-9,2017
17_10_NL EVCCentrumVigor,TheNetherlands March29-31,2017
17_11_SK NárodnýústavceloživotnéhovzdelávaniaNÚCŽV,Slovakia Jan.10-12,2017
Source: PilotDatabase
The11PeerVisitswerecarriedoutinAustria,France,Lithuania,theNetherlands,Portugal,andSlo-vakia.Thedurationofthevisitswasbetween2and2,5days;thedurationofthePeerReviewswasbetween1.75daysand2days(2daysformostPeerReviews)
Allinall,28Peersbecameactiveduringthepilotphase(seealsobelow).All13Peersthatonlypartic-ipatedinonePeerReviewwereemployedasnationalPeers.MorethanhalfofthePeersparticipatedinmorethanonePeerReview(54%),mostofwhomin2PeerReviews(39percentagepoints).15%wereactivein3ormorePeerReviews(amongwhomonepersonfromthecoordinatingbodywith7participations).ThesePeerswithmultipleengagementsallcamefrompartnerinstitutionsandwereactiveduringthewholeproject(participationintransnationalmeetingsandinPeerTraining,respon-sibilityforIntellectualOutcomes).Overall,therewere52deploymentsofPeers;onaverage,aPeerwasactive1.8timesduringthepilotphase.
Table5: NumberofdeploymentsperPeer
Deployments NºPeers %1 13 462 11 393 2 74 1 47 1 4
Total 28 100Source: PilotDatabase
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 14
The Peer Reviewswere conducted in teams of 2 to 7 Peers,with an average of 4.7 Peers per re-view/institution. The standard number of Peers according to the Manual is four, but larger PeerTeamsarepermissible if theorganisationandthePeersagree. Inthecaseofourproject italsoal-lowedmorepeopletoexperiencetheprocessfirst-hand.
Figure6: NumberofPeerReviewswith…Peers
Source: PilotDatabase
ThenumberofPeersinthepilotphaseseemedtobesuitablenomatterwhatthesizeoftheteam,andnorecommendationsweremadetoadjustthenumberofPeersintheManual:Ofthoseprovid-erswith4Peersontheirreview’sPeerTeam,3answeredinthesurveythatitwastheoptimalgroupsize.FromthePeers’perspective,41.7%alsorated4theoptimalgroupsize,another41.7%judgingitworkablebutthatothergroupsizesarepossibleaswell.Theotherratingsofgroupsizedonotdeliv-erstatistical/usefulinformationonadequacyofgroupsize.
All PeerReviewswere carriedout in a transnationalmannerwith at least twoPeers coming fromanothercountry;inallPeerReviewsbutone,theteamconsistedofatleast50%transnationalPeers.Thus inthispilotphasethedeploymentoftransnationalPeers(29or56%ofallPeers) forthefirsttimeoutnumberedthenationalPeers(23or44%).ForanoverviewofthenumbersofPeers(nationalandtransnational)perPeerReviewpleaseseetheAnnex.
3.3 QualityAreas
InallPeerReviews, theEuropeanQualityAreaswereused.As recommended in theManual,mostpilotsreviewedtwoQualityAreas,exceptforonepartnerwhoreviewedthreeandonepartnerwhoreviewedonlyoneQualityArea.
Exceptforonecase,allPeerReviewscoveredatleastone“core”qualityarea(i.e.oneoftheQualityAreas1,2,3,4).Qualityarea8(“QualityManagement”)wasnotchosenbyanyinstitution.FeedbackfrompartnerssuggeststhatthiswasduetothefactthattheQualityArea(whichwascalled“QualityAssurance”)wasmisunderstoodtomeanonlyexternalevaluation(subtitleofearlierversionofthisQualityAreawasmisleading: “Evaluationproceduresand reportson tacticandoperational level”),whiletheindicatorsclearlypointtoamorecomprehensiveviewofqualityassurance.QA1(Identifi-cation)andQA5(Information,GuidanceandCounselling)werereviewedmostoften.Allotherquali-tyareaswereselectedbyatleastoneinstitution.Coverageofthequalityareasisthusoverallsatis-factoryandshowswhichissuesthepartners’institutionsarecurrentlymostinterestedin.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 15
Table7: QualityareaschosenbyVNFILproviders
Code
NºofQAs
QA1Iden
tification
QA2Do
cumen
tatio
n
QA3Assessmen
t
QA4Ce
rtificatio
n
QA5Inform
ation,Guida
nce,
Coun
selling
QA6Stakeh
olde
rCo
ordina
tion
QA7Co
mpe
tencesof
Practitione
rs
QA8Qua
lityMan
agem
ent
QA9Organ
isatio
nofVNFIL
Provision
Special
evalua
tionqu
estio
ns
16_01_NL 2 1 1 ✓16_02_AT 2 1 1 ✓17_03_LT 2 1 1 --17_04_FR 2 1 1 ✓17_05_PT 2 1 1 ✓17_06_AT 2 1 1 ✓17_07_AT 2 1 1 ✓17_08_PT 2 1 1 ✓17_09_AT 3 1 1 1 ✓17_10_NL 2 1 1 ✓17_11_SK 2 1 1 ✓
Total 2,1 6 2 1 3 5 1 4 0 1 10
Source: PilotDatabase
Allbutone institutiontookadvantageofthepossibilitytoposespecialevaluationquestionstothePeers. Some institutions even did so extensively. This contributed to tailoring the Peer Review tospecific information needs and making its outcomes useful to the institution. In some instances,however, the special evaluation questionswere in fact asking for expert consultancy and/orwereimpossibletoanswerintheframeworkofaPeerReview.ThisputstressonsomePeerTeamswhothoughtittheirdutytoanswerthesequestionseventhoughtheManualspecifiesthatrecommenda-tionsandconsultingareonlypartof thePeerReview inexceptionalcases (and thiswasalsohigh-lightedduringthetraining).
3.4 Peers
Peerapplications
In total, 32 people applied online as Peers, 20women and 12men, fromdifferent countrieswithvarying institutionalbackgrounds.Aboutone thirdof theapplicants (10)work forVNFILproviders.About50%ofallapplicantshadexperienceinVNFIL.5oftheapplicantsdidnotparticipateinPeerReviews,4ofwhichhadalsoparticipated(at leastpartly) inthePeerTraining. Intheend28PeersbecameactiveduringthepilotPeerReviews.OneapplicationwasonlypartlyfilledoutandonePeerdidnotfilloutaPeerapplicationatallanddidnotparticipateinthetrainingeither.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 16
BackgroundandcharacteristicsofPeers
Of the Peers, about two thirds (18)were female, one third (9)male. They came from all partnercountries:9fromAustria(almostathird–duetothehighnumberofAustrianpartners),6fromTheNetherlands (including the coordinating body), 5 from Lithuania (with a high number of nationalPeers,4fromPortugal,3fromSlovakiaand(only)1fromFrance.
Figure8: CountriesofPeers
Source: PilotDatabaseandPeerRegister(merged)
Of the28peoplewhoparticipated in theroleofaPeer,19were froman institutionthatwasalsobeingreviewedintheproject(67,9%).TheotherswerePeersexternaltotheprojectandrecruitedbytheProvidersforthePeerReview.
WhetherPeerswereinvolvedinthePeerReviewoftheirowninstitutionwasalsoaquestionaskedinthesurvey;15Peers(64%)answeredinthesurveythattheydidso(whichcorrespondstotheanaly-sisabove)2,twothirdsoftheseasFacilitators,onethirdas interviewee,onePeeralsoasadirectorandtwoPeersalsoinotherfunctions:onesupportedtheFacilitator,theothertrainedPeers(multi-pleanswerspossible).
Figure9: FunctionsofPeersduringreviewofowninstitution
Source: OnlineSurveyofPeers(N=15),multipleanswerspossible
2NotallPeersansweredtheonlinesurvey,seeaboveresponserates.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 17
ThetablebelowshowsthedifferentinstitutionalbackgroundsoftheactivePeers:
Figure10: InstitutionalbackgroundsofactivePeers
Source: PeerRegister,N=27
15ofthe28Peershadexperienceinvalidationofnon-formalandinformallearning,manyofthemalsolong-temexperience(median:7years;mean:9years),6Peersevenfor10ormoreyears.
ThemajorityofthePeershad(some)experienceinevaluation/qualityassurance.Accordingtothein-formation in the Peer application, 20 of the 28 Peers had previously done review/evaluationwork.AboutonefourthofthePeers(7)hadhadtrainingasISOauditor(2alsoasexternalauditor),2PeersweretrainedEFQMassessors.4Peershadhadsomeotherrelevanttraining(otherQMsystems,train-ingduringuniversitystudies),abouthalfofthePeershadhadnotraininginqualityassurancebefore.
Figure11: PreviousQualityAssuranceTrainingofPeers
Source: PeerRegister,N=26
18werefromaninstitutionthathadbeenreviewedbefore,mostlyinISOaudits(orrelatedassess-ments),someinPeerReviewsduringpreviousEuropeanprojects.
Peerswere also asked for a self-assessment of their expertise in different skills and competencesnecessarytocarryoutaPeerReview.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 18
Table12: ReviewskillsofPeers(self-assessment)
Source:PeerRegister,N=26
Accordingtotheirself-assessment,Peersbringaveryhighlevelofrelevantreviewskillstothejob:
§ Beingabletogiveoralfeedback,conductinterviewsandmoderategroupsaretheareaswherethePeersratetheircompetenceshighest,followedbyanalysingqualitativedata,writingreportsandgeneralskillslikeconflictmanagementandtimemanagement.
§ Conductingobservations,analysingquantitativedataanddoingreviewworkinforeignlanguagesareskillswherePeersfeellesscompetent.
§ ThePeershavecomparablylowexpertiseinscientificevaluationsofvalidation.
Thepreviousexperiencesofreviewprocessesorevaluationskillsdidnotinfluencethecompositionof thePeerTeamssinceotherconsiderations tookprecedencewhenthePeerTeamswereput to-gether(seeabove“Peerpuzzle”).ItturnedoutthatsomePeerTeamsdidnotencompassPeerswithreportwritingskills,althoughtheyhad indicatedotherwise in theirapplication,or that insome in-stancesPeerCoordinatorsfeltoverwhelmedbytheirrole.SoperhapssomeofthePeerswereover-confidentintheassessmentoftheirownskills.
(→FormoredetailsonprofilesofPeers,seetablesinAnnex)
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 19
3.5 PeerTrainingandotherformsofsupport
3.5.1 Trainingandsupportoffered
EuropeanPeerTraining,June2016,Vienna
TheEuropeanPeer Training tookplace in theChamberof Labour („AKBildungszentrum“, Theresi-anumgasse 16-18, 1040 Vienna) and at the site of theWeiterbildungsakademie between 13th and17thofJune2016.
The Peer Review Training aims to impart to prospective Peers all necessary knowledge, skills andcompetencestoconductaPeerReviewinaprofessionalmannerandaccordingtothequalitycriteriasetoutintheEuropeanPeerReviewManual.ThetrainingfollowedtheEuropeanPeerTrainingcur-riculumthathadoriginallybeendevelopedinanearlierprojectandhadsubsequentlybeenadaptedtoVNFIL in thecurrentproject.Thetrainingcoveredallphasesof thePeerReviewandputspecialemphasisonthequalityofthePeerReviewprocessandtheprofessionalroleandtasksofthePeers.It isdescribed indetail ina separatedocument.3The trainingalso includedan internationalWork-shopwithAustrianexperts(afternoonofJune15)4andanextensiveplanningphasefortheupcom-ingpilotphase.
Uponcompletionofthetraining,allparticipantsreceivedaEuropeanPeerTrainingCertificatefromEPRAandwereincludedintheEuropeanPeerRegister.
Adetailedagenda, apowerpointpresentationandothermaterialweremadeavailable for furthertrainingsonthenationallevel.
EuropeanPeerTraining,June2016,Vienna
23peopleparticipatedintheEuropeanPeerTraining(ofwhichonepersonwasthetrainerandoneperson participated only partly).5The majority of the active Peers had thus undergone the sametraining(seealsobelow).Sincemostparticipantshadmultiplefunctionsinthepilotphase,thisalsomeantthattherewere1to2peoplepresentfromeveryreviewedinstitution(atleastoneperinsti-tution).3participantsoftheEuropeanPeerTrainingsdidnotparticipateinPeerReviewsafterwards.
3Gutknecht-Gmeiner,Maria(2018):PeerTrainingProgrammefortheProject„PeerReviewVNFILExtended“,Vienna.4Workshop“ValidationofNon-formaland InformalLearning(VNFIL) inaEuropeanPerspective.ExchangeofExperiencesandPeerReviewasanInstrumenttoStimulateQualityDevelopment”organisedwithsupportoftheChamberofLabour.39Austrianandinternationalparticipants.5Theattendancelistthusshows22participants.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 20
NationalTrainingNetherlands,October2016
AnationaltrainingwascarriedoutonOctober5thand6th,2016,intheNetherlands.Thetrainingwasfullydocumented. Itdealtwith themost importantelementsofPeerReview.TheseconddaywasusedtopreparethefirstDutchPeerReview(Libereaux,November2016).
12People,amongwhomallDutchPeerswhobecameactiveinthepilotphase,tookpartinthistrain-ing.ThreeDutchPeersthushadtwotrainingexperiences–ontheEuropeanandthenationallevel.Thetrainingalsoinvolvedfurtherpeoplewhowerenotdirectlyinvolvedinthepilotphase.
DutchPeerTraining,October2016
Otherformsofsupport
Other forms of support camemainly from the coordinating body (see above: 3.1.Organisation ofPilots)andthroughmentoring/learningactivitiesbetweenpartners(cf.MentoringReport).
3.5.2 Participationintrainingsandotherformsofsupport
MonitoringdataandtheresponsesfromthePeerSurveyshowthatalmostallPeersunderwentsomesortoftraining.
Figure13: Participationin(typesof)PeerTrainings
Sources: PilotDatabase,DocumentationofEuropeanandNationalPeerTrainings;OnlineSurveyofPeers;
datamerged;N=28;multipleanswerspossible
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 21
Almosttwothirds(18res.64%)ofthePeershadtakenpart intheEuropeanPeerTraining.6Peers(21%)participatedinaNationaltrainingintheNetherlands,3ofwhichhadalsoparticipatedintheEuropean Peer Training. Therewere seven active Peerswho neither participated in the EuropeanPeerTrainingnortheNationalTrainingintheNetherlands,fourofthesehadhadotherPeerTrain-ings.Allinall,6peoplehadparticipatedinotherPeerTrainingsprevioustothecurrentproject,twoofwhichhadtakenpart intheEuroPeerguidtraining,whichfollowedthesamebasicqualitystand-ardsandcurriculum.
ThreePeershadreceivednotrainingatall (11%),buttheydidreceiveotherformsofsupport.Theresthadatleast1PeerTraining(71%)oreventwo(18%),mostofwhichparticipatedintheEuropeanPeerTraininginJune2016inVienna.ThisshowsacomparablyveryhighleveloftrainingofPeers.
Figure14: NºofPeerTrainingsperPeer
Sources: PilotDatabase,DocumentationofEuropeanandNationalPeerTrainings;OnlineSurveyofPeers;
Datamerged;N=28
ThemostcommonandmostimportantformofsupportforPeerswassupportfromthecoordinatingbody(2/3), followedbypreparatorymeetings(50%)withtheotherPeers intheirteam,4ofwhichmentioneddoing itoverSkype. In2of thesecases, theVNFILproviderand/or the facilitatorwereinvolved.ThecoordinatingbodyreceivedsupportfromtheEuropeanPeerReviewAssociation.
InternalmentoringwasreportedasaformofsupportforathirdofthePeers(allofwhichworkedinaninstitution thatwasalsobeing reviewed).Only twoPeers (8%)hadnot receivedany furthersupportduringthepilotphase,theyhad,however,participatedintheEuropeanres.DutchPeerTraining.
Figure15: OtherformsofsupportforPeers(multipleanswerspossible)
Source: OnlineSurveyofPeers(N=24,multipleanswerspossible)
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 22
Except for the LithuanianVNFILprovider (whowas, however, supportedby the Lithuanianprojectpartner), all reviewed institutionshad staff trainedduring theEuropeanPeerTraining.9of the11providers (82%) had support from the coordinating body; 4 did some internal mentoring/mutuallearningactivitiesduringthepreparatoryphase(seealso→Mentoringreport)and4providersmen-tionedhavinghadapreparatorymeeting.Altogether,allproviders received somekindof support;54,5%receivedmorethanoneformofsupport,2onaverage.
3.5.3 Feedbackontrainingsandsupportduringpreparatoryphase
FeedbackonEuropeanPeerTraining
TheEuropeanPeerTrainingwasconsideredeither“helpful”(38,5%)or“veryhelpful”(61,5%)byall13Peerswhoansweredthisquestion.
Providerswerealittlelesssatisfied–butthetrainingwasalsonotgearedtowardsthem,buttothePeers.Still,almostallproviders(91%)foundtheEuropeanPeerTrainingeither“useful”or“veryuse-ful”aswell,exceptforonewhofoundit“notsouseful”,thereasonbeingthatthestaffofthispro-viderwerealreadyveryexperienced.
Table16: FeedbacktoEuropeanPeerTraining
Howhelpful? Peers Providers
Veryhelpful 61.5% 50%
Helpful 38.5% 40%
Notveryhelpful -- 10%
Nothelpfulatall -- --
Source: OnlineSurveyofPeers(N=13)OnlineSurveyofProviders(N=10;oneproviderdidnotsendstafftotheEuropeanPeerTraining)
BeingtakenthroughallthestepsofthePeerReviewand/orpracticingtheminsimulations/exerciseswerementioned as particularly helpful aspects. 5 participants (out of 9who commentedonwhattheymissedorwhatshouldbechanged)expressedthewishforevenmorepracticalexamples/casestudies.One Peer commented: „Imissed information on how towrite a Peer report. The formatprovidesalotofinformation,butdoesnotprovideanyexamples.[...]Ithinkitwouldbeagoodideatoadd'writingthePeerreport'tothetraining(mainlythepartwhereyoudescribeyourfindings,therestisprettyclear).“GettingtoknowtheotherPeerswasalsoapositiveaspectoftheEuropeanPeerTraining(asintended).
The(mostlyactivatingandexperience-oriented)trainingmethodsusedwereconsideredeitherhelp-ful(onethird)orveryhelpful(twothirds)bythePeers.
TheEnglishlanguageaffectedparticipationfor5participantsinsomeinstances,therest(12)experi-encednoproblemsatall.
Feedbackonandothersupport(includingNationalTrainingintheNetherlands)
13Peersalsogavefeedbacktotheothersupport(includingtheDutchnationaltraining):thesupporttheyreceivedwasratedeither“helpful”(38,5%)or“veryhelpful”(53,8%)byallparticipantsbutone,whoratedit“nothelpfulatall”butdidnotfurthercommentontheirrating.CommentsinthePeersSurvey suggest that for some participants, a littlemore time for further elaboration and practicewouldhavebeenhelpful(especiallyforthosewhohadnotbeenintheEuropeanPeerTraining),butoverallsatisfactionseemedhigh.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 23
3.5.4 Levelof“preparedness”ofVNFILprovidersandPeers
91%(10outof11)ofprovidersratedtheirinstitutionas“wellprepared”or“verywellprepared”forthePeerReview.Theoneinstitutionthatfelt"notsowell"preparedansweredthatthiswasduetoconductingthefirstPeerReviewintheproject.
ProvidersfeltthatPeerswerewellprepared(45,5%)orverywell(54,5%)preparedfortheirtask.ThePeers themselves felt slightly lessprepared (44%verywell preparedand44%well prepared) thanindicatedbytheproviders6:Butonly3Peersfelt“notsowellprepared”,twoofwhichwerenewtoPeerReview,andhadsomeexperiencebuthadto“jump into” the firstPeerReview in theroleoffacilitator(andactuallyfeltbetterpreparedwhenfirstactiveasPeerlateron).Oneresponsepointedout that level of “preparedness” also has to dowith howmuch time and energy the Peers them-selvesinvestedinthepreparationofthePeerVisit(byreadingandanalysingtheSelf-Report,prepar-ingtheVisitetc.).
Table17: Levelof“preparedness”ofthePeers
HowwellpreparedwerethePeers?
PerspectiveofPeers PerspectiveofProviders
Nº % Nº %
Verywell 11 44 6 54.5
Well 11 44 5 45.5
Notsowell 3 12 -- --
Notwellatall -- -- -- --
Sources:OnlineSurveysofPeers(N=25)andProviders(N=11)
3.6 PreparationofthePeerReview(Phase1)
3.6.1 PreparationoftheVNFILProviders
TasksofpreparationofthePeerReviewwereclearlydistributedattheVNFILProvidersastheopenanswersofthesurveyshow.ThemainpreparationworkhadtobedonebythePeerReviewFacilita-tor.Mostly s/heworked closelywith a teamwhere e.g. themanager or the qualitymanagerwasinvolved. S/hewas responsible for doing the agenda,writing the self-reportwith definitionof thequestionsandthechoiceofinterviewees.Severalpeoplewereinvolvedinthepreparation,especiallythedirectorindrawinguptheself-report.FortheconcretepreparationofthePeerReviewmeetingswereheldwiththepersonsinvolved.Sotopicsandquestionsofthecounsellorswereconsideredintheself-report.
Asmanyaspossiblewereinformedabouttheactivitieswithintheinstitution,aboveallthecounsel-lorsandassessors. Informationtookplaceindifferentformslikepresentationsindenregularteammeetingsorviae-mail.
3.6.2 Self-Report
All Self-Reports followed theprescribed format. Peerswere asked in thequestionnaire if the Self-ReporthadbeenagoodbasisforthePeerReview(i.e.ifitwasunderstandableandprovidedallrele-vantinformation);60%answered“yes”,40%answered“partly”;nooneanswered“no”.
The qualitative answers showed that some Self-Reportsweremore, some lessmeaningful for thePeers:Someanalyseswerenotveryextended,sothePeerTeamhadtoaskquestionswhichcouldhavebeenansweredalreadyintheSelf-Report;insomecases,thereportwasdeliveredverylate(oreventoolate)tobeusefulforthePeersinthepreparationoftheReview.Itwasstatedininterviewsthat the Self-Report should includemoredetaileddescriptionof theprovider’sVNFILprocessor a
6ThisisaphenomenonthathasbeenobservedalsoinpreviousPeerReviewpilots,intheirself-assessmentsPeerstendtobemorecriticalthantheproviders.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 24
descriptionof the rolesof involvedpersons inVNFIL-processes.Thequalitativeanswers show thatthePeersrealisedduringtheexperiencehowimportanttheSelf-Reportis(cf.alsoManual)andthatitshouldbewritten inawaytocatertooutsiderswhodonotknowthe institutionorthenationalsetting. As always, therewere somewhowantedmore detail and otherswho thought the reportshouldbeshorter.
TheextentandthecontentofthePeerreportswasveryvarious.Youcouldreallytellthatsomepro-vidersmadea lotofeffortandthatsomeprovidersdidnothavetimetopreparethe[self]report.One of the reportswas a one-on-one copy from another provider.Most of the time, thereweresomequestionsafterreadingtheSelf-Reports.DuringaSkypemeetingbeforethevisit,theseques-tionswerealwaysansweredbythefacilitator.Ingeneral,Inoticedthatthe'simplestuff'isforgottenbythefacilitator.Justbecause(s)heisveryintotheprocess,andforgetstomentionthesmallthingsthat are very crucial for thePeers tounderstand theprocess.Agoodand completeSelf-Report isveryimportant,especiallywhenitconcernstransnationalPeerReviews.WithoutagoodSelf-ReportandagooddescriptionoftheVNFIL-provisionintheparticularcountry,asaPeerTeam,youneedalotof(valuabletime)tounderstandthisandbeabletocarryoutagoodPeerVisit.(experiencedPeerandFacilitator)
3.6.3 PreparationinthePeerTeams
SeveralpartnersemphasisedintheinterviewsandattheworkshopthatjointpreparationinthePeerTeamwasveryimportant.ItwasconfirmedasgoodpracticethattheteammeetsintheafternoonofthefirstdayofthestaytodiscusstheSelf-Reportandtoprepareinterviews.Atthispointoftime,thetasksmustbedividedaswell.Inaddition,intensiveexchangebye-mailandSkypemeetingswereusedforpreparationbeforethePeerVisit.ItwasidentifiedasbestpracticeforthePeerTeamstoworkouttheinterviewquestionsinadvancesothereislesstimepressureontheteamimmediatelybeforethePeerVisit.
ThefindingsofthepartnerworkshopalsoconfirmedthatatraininginPeerReviewforallprospectivePeersisaprerequisiteforasuccessfulPeerReview.Peersmustbeabletoconductinterviews,ana-lyse,drawconclusionsandwriteareportaboutthefindings.
AsrequiredbytheManual,thePeerVisitagendashouldbediscussedinadvancebetweenproviderandPeerssothattheycanmakesomesuggestionsforimprovement.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 25
3.7 PeerVisit(Phase2)
3.7.1 PeerVisitAgendas
The structureof theagendasof thePeerVisits largely followed the recommendationsof thePeerReviewManual;mostadaptedthemodelagendafromtheToolbox.ThePeerVisitsonlyvariedslight-ly in length,followingtheexperiencesoftheprojectEuroPeerguid-RVCthevisitstooktwodays(oralmosttwodays).TheintensityofthePeerVisitshowevervariedwith4PeerReviewshavingfewerthan10datacollectionsessions,andtheothersupto12oreven14sessions.
Table18: PeerVisitAgendas:Numberofsessions,numberofinterviewswithdifferentinterviewgroups
PeerVisit Nºofinterviews
Other(e
.g.Presentation)
Observatio
ns
Tourpremise
s
Documen
tana
lysis
Totalnºofse
ssions
Overalltim
efor
analysis(hou
rs)
Qua
lityAreas
Code
Duratio
nda
ys
Prep
arationda
ys
Man
agem
ent/coo
r-dina
tors
Coun
sellors/A
sses-
sors(Traine
rs)
Cand
idates
Form
ercan
dida
tes
Otherstakeh
olde
r
Otherstaff
Nºofinterviews
16_01_NL 2 0,25 2 3 1 0 2 4 12 1 1 0 0 14 4,5 4;516_02_AT 2 0,5 3 2 1 2 1 0 9 0 1 1 1 12 6,25 6;117_03_LT 1,75 0 2 2 ? 1 0 0 5 1 1 1 0 8 5,5 1;717_04_FR 2 0 2 2 ? 1 1 0 6 0 0 1 0 7 3 1;517_05_PT 2 0,5 2 7 2 0 0 0 11 0 0 1 0 12 7 4;717_06_AT 2 0,25 4 0 2 0 0 1 7 3 0 1 1 12 5,75 3;517_07_AT 2 0,5 2 2 2 2 2 0 10 0 1 1 0 12 6,75 1;217_08_PT 2 0,16 2 7 2 0 0 0 11 0 0 1 0 12 6,75 4;717_09_AT 2 0 2 5 2 0 0 1 10 0 0 1 0 11 6 1;2;517_10_NL 1,75 0,25 2 1 ? 1 0 2 6 1 0 0 1 8 4,5 7;917_11_SK 1,75 0,4 1 1 2 1 1 1 6 2 0 1 0 9 0 1;5Average 1,9 0,2 2,2 2,9 1,3 0,7 0,6 0,8 8,5 0,7 0,4 0,8 0,3 10,6 5,1 2,1
Source: PeerReviewVNFILExtendedPilotDatabase,informationfrompilotdocumentation,esp.Agendas,PeerReviewReports
Inclusionofstakeholders
AnanalysisofthePeerVisitAgendasshowsthattherelevantstakeholdergroupswerebyandlargeincluded in the Peer Visits. A very positive result is that partners alsomanaged to involve formercandidates.Inonecase,anonlinesurveyonthePeerReviewtopicwascarriedoutwithformercan-didatesbeforethePeerVisitandtheresultsweremadeavailableforthePeers.
Thereare,however,somePeerVisits,forwhichtheofficialagendashowednointerviewswithcan-didates and in one instance also not with practitioners. During the partnermeeting in Bratislava,partnerspointedout that in fact thesegroupshadbeen included (e.g. throughan interviewtapedbeforethePeerVisit).Partnerswereaskedtofurnishfurtherinformation,buttonoavail.Withdatamissing, there canbeno final assessmenton thequestionof full inclusionof candidates from thedocument analysis. Yet the feedback from Peers and providers in the survey suggest that by andlargecandidateswereinvolvedsufficiently(seebelow).
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 26
Datacollectionandtimeforanalysis
Themost frequentdata collectionmethodwas interviews,manyof themwithone intervieweeoronlyaverysmallgroupofinterviewees.Insomecases,staffmembersoftheprovidersgavepresen-tations.Inatleast4casesobservationstookplaceandin3casesalsoadocumentanalysis.
TherewasmoretimeallottedtoanalysisthanintheEuroPeerguid-RVCprojectwiththeAustrianandPortugueseagendasclearlyinthelead,whileintheNetherlandsandinFrancetimeforanalysisdoesnotseemasimportant.Perhapswealsoseeculturalpreferencesinthisrespect.
AllVNFILprovidersanswered in thequestionnaire that thePeerVisitproceededasplanned in thePeerVisitAgenda.But itwaspointedout in theworkshop, that in somecases therewas too littletimescheduledforreflectionandforinterviews.
Involvementofcandidates
Thepilotphaseshowed that itwaspossible to recruit candidates for the Interviewsaspartof thePeerReview.ProviderswerecontentwiththerecruitmentofcandidatesforthePeerVisit,thePeerswerealittlemorecritical.TheresultsshowthatallProvidersmanagedtoincludecandidates.
Table19: RepresentationofcandidatesinPeerVisit
Howwellwerecandidatesrepresented?
PerspectiveofPeers PerspectiveofProviders
Nº % Nº %
Fullyrepresented 9 37.5 5 45.5
Sufficientlyrepresented 15 62.5 6 54.5
Notsufficientlyrepresented -- -- -- --
Notatallrepresented - -- -- --
Sources:OnlineSurveyofPeers(N=24)andProviders(N=11)
Accordingtotheprovidersitwaspartlyachallengetofindcandidateswhohavetimeduringtheday,sincemost are in employment. It is thereforenecessary to show flexibility and fix interviewswithcandidatesintheprocedureaccordingtotheirtimepossibilities.
Anotherissueisthatusuallyonlycontentandcommittedcandidateswhoarecommittedtothevali-dationprocessareavailableforinterviews.This,however,doesnotpromoteabalancedviewofthevalidationoffer.Most important,possibledrawbacksandstumblingblocksmaynotbedetectedbyonlytalkingtothesuccessfulcandidates.
Itwouldbehelpfultomeet"dissatisfied"participants,or"dropouts"fromtheprocess.Itisdifficulttoorganise,butitwouldpreventpotentialbiasfromself-selection.(OnlineSurveyofPeers)
Inordertoreachthosecandidateswhohave(temporarily)discontinuedtheVNFILprocess,apartnerinstitutiondevelopedanonlinequestionnaireandsent it tothosecandidateswitharequesttore-turnitcompleted.ThequestionsconcernedtheirexperienceswithVNFIL.Thisapproachworkedwellandcouldbe includedinthePeerReviewManualtoprovidethePeerswithadditional informationaboutthislesssuccessfulgroupofcandidates.Thisis,however,adatacollectionactivitythatshouldtakeplaceduringself-evaluation(aswasthecaseintheexamplecited).
Anotheravenueforcapturingtheexperienceoftheunsuccessfulcandidatesisthroughthepercep-tionoftheVNFILpractitioners:
Someonewho reportsonhisown failure is certainlydifficult to find. It is alwayseasier toget thesuccessful ones.But through the coaches and theassessors you canapproach thedifficulties too.Theyhaveagoodoverviewofwhatworksandwhatdoesnot.(Source:Interview)
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 27
Opennessandprotectionofinterviewees
Protectionofanonymityofintervieweescanbeaproblemifinterviewgroupsare(very)smallaswasthecaseinmostPeerReviewsinVNFILand/orifonlyonerepresentativeofagroupofstakeholdersisinterviewed.Thelattershouldbeavoidedifpossible.Generally,ProvidersundergoingPeerReviewaswellasPeersshouldbemadeawareofthisproblem.Duringthepilotphase,theopinionsofindi-vidualweregiven in thePeerReviewReportsnaming the source.This isprincipallynotadmissibleandwouldrequireofficialconsentbytheintervieweeconcerned.Obviously,thegeneralfeelingthateverybodyis“friends”inaninstitutionobfuscatesthefactthatthisbehaviourisaseverebreachofprofessionalconduct.
Whenaskedaboutpossiblereticenceofinterviewees(especiallystaffwhomightbeafraidofreper-cussions),Peersmostlyreportedthatallinterviewshadbeenconductedinaveryopenmannerandthat intervieweeshadbeen forthcoming. In rare cases,where interviewparticipantshadhad littleinformationonthePeerReviewbeforehand,therewassomereluctanceatfirst.Inthesecases,thepurposeoftheinterviewandtheconfidentialityoftheinterviewswereexplainedandintervieweesbecamemorerelaxed.
3.7.2 ExperiencesduringPeerVisits
PeerssaidininterviewsandattheworkshopthatitisveryimportantthatallrelevantdocumentsareinplaceatthestartofthePeerVisit.InadditiontothosethatwerepreviouslypreparedliketheSelf-Report,uponrequestofthePeersadditionaldocumentsmaybefurnished,likeanexampleofaport-foliooranoverview from feedbackof thecandidates.Peers shouldask for it ingood timeso thatthesedocumentsandalsonecessaryinformation-e.g.abouttheproceedingsortheinterviewees-canbeprepared.
InatransnationalPeerTeam,it isessentialtoreceiveinformationabouttheeducationsystemandthe VNFIL system in the country beforehand. Thiswaswhy country ficheswere developed in theproject(O1),butitseemsthatpilotphaseparticipantsdidnotextensivelyusethem.Iftheproviderdoes not deliver the necessary documents in time or if they do not contain enough information,problemswithtimeandschedulemayariseduetoopenquestionsandmisunderstandings.
ItwasstressedbythePeersthataconstructiveworkingatmospherebetweenallpersonsinvolvedisveryimportant,whichwasthecaseinallTeams:Morethan83%ofthePeersratedthecooperationinthePeerTeamas“verygood”,therest(17%)as“good”.Noconflictswerementioned.
Table20: QualityofcooperationinPeerTeam
CooperationinPeerTeam Nº %
Verygood 20 83.3
Good 4 16.7
Someconflicts 0
Notgood 0
Total 24 100
Source: OnlineSurveyofPeers(N=24)
ThegoodcooperationbetweenPeersis,ifnothingelse,theresultofagoodjointpreparation.Aposi-tive effect is achieved by communicating openly and transparently between Peers and the repre-sentativesof the institution. Itmustbe clear from thebeginningwhowill takeonwhich tasks.Asconfirmed in the interviews, Skypemeetings for arrangements in advance andpreparationon thefirst evening serve this purpose.During thesemeetings, the agendaand interviewquestionsmustalsobediscussed.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 28
According to theprojectpartners, timepressurehasanegativeeffecton theatmosphereand theresultsof thePeerVisit.ProviderandPeershavetotakecareof thetimemanagementduringthePeerVisit. Iftoomuchtimeisused,forexample,ontheinterviews,thistimewillbemissingintheimportant phases of reflection and drawing conclusions, which are paramount. A lack of time forreflectionwascriticisedbyseveralPeers.Theproblemseemstohavebeenaminorone,though:Inthesurvey,50%ofthemsaidtheyhadexperiencedsomeproblemsbecauseoftimepressure,thoughnoneofthemwereseriousproblems.Mentionedissueswerethat ittookmoretimethanplanned,sometimesduetoextratranslationtime.Theother50%foundthetimeforthePeerVisitsufficient.
ThePeershadtheexperiencethatitisnecessarytoworkdigitallywithlaptopsduringthePeerVisit.Therefore,PeersshouldalsobegrantedaccesstoWi-Fiandaprinter.OnePeerdescribedthattherewasadelayduetoanunclearroomsituationfortheinterviews.
SomePeerTeamsexperiencedlanguageproblems.Over50%ofprovidersansweredinthequestion-nairethatEnglish/languageproblemswereanissue;thoughtheywerenotnecessarilyalwaysprob-lematic.AllPeerswerefluentinEnglishaswellasintheirmothertongue.Aprobleminthiscontextwas that somemanageror interviewees in the institutionsdidnot speakEnglish.Notonlydid thiscausealotoftimebeingspentontranslations;insomecases,thesummarisingnatureofthetransla-tions,whichweredonebynationalPeers,alsoleadtolossofcontent.Also,spontaneousquestions,comments andnotesbecamemoredifficult.Additionally, consecutive interpreting canbe veryex-haustingforthepersoninroleofthetranslator.Consequently,somePeerswhodidnotunderstandtheoriginaltextdidnotfeelasinvolvedintheinterviews.Forsuchcasesatimely,directtranslationisrecommendedintheinterviews,whichmeansnottoomuchshorteningoftheoriginaltext.PossiblelanguagedifficultiesaretobedeterminedalreadyaheadofthePeerVisitduringandhavetobetak-enintoaccountfortheschedulingofthePeerVisit.
Peers said that for successful interviews well prepared questions are crucial, as well as the rightchoiceof interviewees. In somePeerReviews, thequestionswere compiledat shortnoticeat thebeginningofthePeerVisit.
Both the preparation of interview guidelines in advance and comprehensive information of inter-vieweesatthebeginningofinterviewsarepartofthePeerReviewprocedureandweretaughtdur-ing thePeerTraining.Balancing thenarrative flowof interviewswhilenot losing sightof themainquestionsoftheinterviewisaskillPeersneedtodevelopfortrulyexploratoryinterviews.Peerswhodonothavetheexperienceareadvisedtostickmorecloselytotheinterviewguidelines.
Peersalsopointedoutthatinsomecasestheinterview(s)withthemanager(s)is/aresocrucialtothePeerReviewfindingsthatit/theyshouldbeattendedbythewholePeerTeam.However,thisshouldnotbegeneralisedsince itwilldependon theconcretesituationand theworkingapproachof thePeerTeam,whichcanhavemoreorlessdistributionbetweenPeerTandems.
AttheendofthePeerReview,apresentationoftheresultsistobeplannedtogetherwiththepro-vider. It is important that this presentation is scheduled bymanagement and that other relevantemployeeslikeacounsellororthequalitymangerarepresentaswell.
IntheexperienceofthePeers,thispresentationfulfilstwomainfunctions:First,itisamotivationforPeerstopresenttheresultsoftheanalysisinmutualappreciation.Secondly,thereviewedinstitutioncanusetheresultsandtheprocessofimplementationcanbegin.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 29
3.8 PeerReviewReport(Phase3)
3.8.1 Feedbacksessionandusefulnessoffeedback
63.6%ofproviders foundthe feedbacktheyreceived in the feedbacksessionandthePeerReviewReport“useful“;theother36.4%founditatleast"partlyuseful”.
3.8.2 Timelinessofreports
Result orientationandgood cooperation inPeer Teamduring the visit is aprerequisite for a solidfeedbacksessionduringthevisitandameaningfulPeerReviewReport.
SomePeerReviewReportsweredeliveredaftertheduedate,butallweredeliveredbeforeprovidershadtofilloutthequestionnaire.
3.8.3 Writingthe“PeerReviewReport”
ThewritingofthePeerReviewReportwasseenasakeyelementincompletingaPeerVisit.Howev-er,somePeersexperiencedproblemswritingthereport.
OneparticularproblemwasthatinsomePeerReviewsitwasdifficulttodealwiththereportstruc-turewhich focusesonQualityAreas. Thiswasdue to the fact that thePeerReviewprocessoftenveeredfromtheoriginalQualityAreasand/orthatfindingswererelatedtomoregeneralissuesandtoucheduponmorethanoneQualityArea.Inaddition,inmanyPeerReviewsalotofemphasiswaslaid on providing direct recommendations and advice for the reviewed institution (which is not acommonpartinaEuropeanPeerReview).ItwasthereforediscussedintheprojectteamtoexpandthePeerReviewReportandadd(sub)chapters forgeneral feedbackandrecommendations/advice.Sincethisis,however,notaconstituentpartofthePeerReviewandwouldmisleadfutureusersinbelieving that giving general feedback and providing advice is amandatory task itwas decided toleavetheformas it is. InfutureroundsofPeerReviewsitcould,however,bepointedoutthattheformcanbeadapted, ifnecessary, as longas thecorepartsare retained. In theend,experiencedreportwriterswillfindawayofincorporatingallimportantinformation.
AnothersuggestionwastoprovideyetanotherguidelineonhowtowriteaPeerReviewReport.Yet,thereportformalreadyprovidesampleguidanceanditisquestionablewhetherlackingreportwrit-ingskillscanbebuiltbyprovidingsuchaguideline.Intheend,thisfeedbackaswellasinformationonthereportingprocessesforsomeofthePeerReviewspointstoselectingPeerTeamsmorecare-fullyandmakingsurethatat leastonmemberhassufficientexperience inreport-writing.Thesug-gestiontoschedulemoretimeduringthePeertrainingtodiscussthewritingofthePeerVisitreportcanbeheeded in futuretrainings. Itwill,however,notremedythesituationof lackofexpertise inthePeerTeam if itdoesnot includean intensive (andtime-consuming)practicalexercise in reportwriting.ThisinturnwouldnecessitatesimulationofanentirePeerVisitandreportwriting(Phase2and3ofPeerReview).
3.8.4 PeerConsulting
Peershadengagedinaconsiderableamountofconsulting–tryingtobeashelpfulaspossibleandanswering“evaluationquestions”ofprovidersasking foradviceon“tricky”problems.Thiswas forthemostpartbeyondthescopeofaPeerReview(seeabove).Duringthepilotphase,waysfordeal-ingwiththissituationdevelopedwiththehelpofthecoordinatingbody–delineatingtheboundarybetweenthekindofadviceor“suggestions”Peerswereablytofurnishandtherequestsforconsult-ing thatPeerscouldnotprovide. In futurePeerReviews it shouldbe (evenmore)highlighted thatspecific recommendations (consulting) are not the aim of a Peer Review (which is an evaluation).Insteadthereviewedinstitutionshouldfinditsownsolutionstoproblems–andnotposeevaluationquestionsthatareimpossibletoanswerinaPeerReview.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 30
3.9 Puttingplansintoaction(Phase4)
Inthesurvey,90.9%oftheprovidersstatedthattheywillactatleastpartlyonthefeedbackreceived(including4providerswhosaidwouldactonalmostallfeedback).Onlyoneinstitutionanswerednotactingupon feedback, the reasongivenwasnothaving receivedany feedbackduring themeetingbut only with the report – so this answer clearly resulted from amisinterpretation of the surveyquestion.
TheresultsofaPeerReviewarefindingsondifferentsub-aspectsofthechosenQualityAreas.Whichtopics are quickly implemented in the institutions depends onwhether or not they are urgent orsilentquestions.Somecannotbeimplementedimmediatelyduetomissingstructuralrequirements.However, thereare someexamplesof findings thatwerequickly implemented, inparticularwhentheywereintherealmoftheVNFILprofessionalsanddidnottouchuponlargerstructures:
§ The recommendation to prepare a set of standard keywords in the context of labourmarketpolicyorinthecontextofacompanyforusingitduringvalidationorcompetenceassessment.
§ Another recommendation that has been implemented concerns the sustainable effect of thecandidate's validation: So that the participants should also later be reminded of their compe-tencestheparticipantswritinge-mailstothemselvesduringthevalidationthattheyreceiveatalaterpointintime.
§ Inonecase,itwasrealisedthattheportfolioshouldnotonlybeusedasatoolforvalidation,butthatitshouldalsobeusableforpreparingcandidates’curriculumvitae
§ Inanothercase,anopenproblemfromapreviousevaluationprocesswassolved:Thisquestionwaswhymanycandidatesdonotcompletethevalidation.AsaresultofthePeerReview,aclos-ersupportfortheparticipantsandanoptimisedschedulewillbeoffered.
Inadditiontothe implementationofconcretemeasureswithinthe institutions,theprojectalsoal-lowed theexchangeofexperience,evenacrossborders. In the interviews, the followingmeasureswereidentifiedexemplary,thatcouldbetransferredconcretely:
§ Itseemedinteresting,thatinonecountryexisttheroleoftheeditorwhoformulatestheportfo-liosoftheparticipantsinsuchawaythattheybecomecompatibletotheNationalQualificationFramework.
§ Itwasalsonoticedthattheroleofinitialconsultationisveryimportantinonecountry.Theap-propriatedegreesare identifiedfromacatalogueof18.000possiblequalifications. Inthisway,thecandidates’competenciesandwhatispossiblealongthequalificationstandardsarematched.
3.9.1 Communicationofresults
At the end of each Peer Visit, the Peers presented their key findings to the involved staff of theVNFIL-provider.Thefeedbackwasverballycommunicated,mostly inthecontextofapresentation,tothedirectorandtheinterviewedstaff.Insomecases,additionalmembersofthestaff,whowerenotinvolvedinthePeerVisit,werepresent,e.g.thequalitycoordinator.Theotheremployeeswereinformedaboutthefeedbackandafollow-upcarriedoutindifferentwaysofcorporateinformationand communication like websites, newsletters or personal information. In single cases it was notpossible that the optimal selection of employees could be informed in the context of the finalpresentation.
Abouthalfofprovidershadeitheranextrapresentationorinvitedotherstafftothefinalmeeting(orboth).Othersonlysharedorpublishedthereport(sometimestranslated).Inthreeinstitutions,therewasnofurtherdisseminationofresultswithintheinstitutionexceptforsharingitwiththedirector.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 31
3.10 Cooperation,rolesandtransnationalaspects
3.10.1 QualityofcooperationduringPeerReviews
TheonlinesurveysforVNFILprovidersandPeersandthequalitativeinterviewsprovideinformationonthequalityofcooperationduringthewholePeerReviewprocess.Cooperationwasgenerallyrat-edverypositively:
Allprovidersansweredthatcooperationwithintheinstitutionhadbeeneither“verygood”(>60%)or“good”.RegardingtheCooperationbetweenprovidersandPeerTeams,bothsidesratediteither“verygood”(>60%)or“good”aswell.Providerswerenoticeably(even)morecontentthanthePeers.
Table21: QualityofcooperationbetweenProviderandPeersCooperationbetweenProvidersandPeers
PerspectiveofProviders PerspectiveofPeersNº % Nº %
Verygood 9 81,8 14 60,9Good 2 18,2 9 39,1
Someconflicts -- -- -- --Notgood -- -- -- --
Total 11 100 22 100
Source: OnlineSurveyofProviders(N=11)andPeers(N=22)
CooperationwithinthePeerTeamswasalsoratedhighlypositive:83.3%found it“verygood”, therestfoundit“good”(seealsoabove).Noconflictswerementioned.
3.10.2 BenefitsofmultipleparticipationofPeersandfacilitators
MultipleengagementswereregardedhighlypositivebythePeers(85%rateditas“mostlypositive”)andnonegativesideeffectswerementioned.CommentssuggestthattheexperienceofPeerswhohadalreadyparticipated inotherPeerReviewswashelpful to theprocessandtothosePeerswhodidnothaveanypreviousexperience.
Experiencingbothperspectives-intheroleasaPeerandasfacilitator-wasalsojudgedaspositiveinthequalitativeinterviews.Inparticular,itwasseenasanadvantageiftherolePeerisfirsttaken.TheinsightintothePeervisitprocessmakesiteasiertomakepreparationsintheowninstitution–alsoonaverypracticallevel:
“Basedonour initialexperience, Ihavedevelopedamatrix thatwaspracticableduringthesubse-quentPeerVisits.”(Source:Interview)
AmutualPeerReview(i.e.providersexchangePeers)isnotdeemedadecisiveelementofthemeth-od, but was rather due to organisational and financial considerations. Thematic expertise of thePeersremainsthemaincriterionforchoosingPeers.AsapeculiarityofthemutualPeerReview,anintervieweementioned that there is a special commitment between the institutions and that thisraisesthemotivationandconstructivecooperationofPeersandprovidersandsoalsothequalityoftheresults.
3.10.3 LessonslearnedfromnationalandinternationalcooperationduringPeerReview
The Peer Reviews offered the institutions the opportunity to get in touchwith other experts in asimilar areaofwork,whichwashighlyappreciatedbothnationally and internationally:On thena-tional level, Peers gained experience in other educational subsystems; internationally, completelydifferent education systems could be explored. The special feature of Peer Review is to gain verydeepinsightintothesystems.Thiswasmainlysupportedbypersonalcontact,especiallythroughthePeerVisits,whereyouintensivelydealwithdetailedquestionsoftheinstitutioninvestigated.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 32
Astransnationalparticipants,thePeersexperiencedthatacloserelationshipwiththevisitedinstitu-tionwas felteven though theywere inadifferentenvironment.7The reasonwas thatyoualreadyknewa lotabout thepartnerand thatyousharedgoalsandworkexperienceswith the institutionyouvisited.Thiscomparabilitysurprisedwhengettingtoknowothereducationalsubsystems.AvisitwithoutcooperationinaPeerReviewwouldnotrendersuchadeepinsight.However,withregardtotransnationalPeerReviews,itisnecessarytomaketherestrictiveobservationthatasaninternation-alPeer,youmusteitherhavepriorknowledgeof the foreigneducationsystemand the institutionyouarevisitinginordertounderstandtheinnerworkingsofthereviewedinstitutionoryourfellowPeersmustbringyouuptoparduringpreparationtime.
Inthiscasetheinternationalpartwasveryinteresting.ButbeforeyoucandoaninternationalPeerVisit, youhave toknowabout thecountry, the systemand the institutionyougo to.Otherwise itcostsalotoftimereally[learning]aboutthatsysteminthewholecountryandthesysteminthein-stitution. It is a lot of information and you need a clear goal with the international partners.(Source:Interview)
3.10.4 Transnationalaspects:TheimportanceofinternationalPeers
In the surveys, the transnational aspects were generally rated positively by all VNFIL providers –morethan80%foundthempositive,therest“partlypositive(withsomeproblems)”.Requirementsres.recommendationsforincludingatransnationalPeerwere
• awarenessoflanguageissues(56%),• sufficientknowledgeofnationalsystem(33%),• sufficientfunding(22%).
Theimportanceofinsightintoeducationsystemsofthecountryvisitedwasemphasisedalsointheinterviews(seeabove).
The opinionwas expressed in some interviews, that Peer Review supports the implementation ofVNFILwellandindifferentways.EspeciallyinatransnationalPeerReview,itbecomesclearthataninternationalPeerhasnoconflictofinterestandcanthusgiveveryopenfeedbackwithouthavingtodealwithissuesofcompetition.Intheviewofonepartner,workingintheforeignlanguageEnglishalsocreatesthenecessarydistancefromeverydaybusiness.The internationalityalsobringswith itan“auraofexpertise”(thosewhocomefromfarawaymusthavesomespecialexpertise…)thatcanstrengthenthemotivationintheinstitutiontotakethefeedbackseriously.Youalsoslipintoanotherrolewiththejourneyandleavethefocusonyourownsystembehind.TheinternationalPeerbringsnoprejudicestothetask,whichhelpstotakeastepbackandlookatthesituationinthevisitedinsti-tutionwithaprofessionaldistance.
Bycontrast,atthenationalPeerReview,especiallyinasmallcountry,theselectionofPeersmustbemadecarefully.AnationalPeerisratherpartofhis/hersystemandconstantlytakesintoconsidera-tionhis/herownpositionwithinthenationalsystem.Cross-connectionsandrelationshipsareauto-maticallytakenintoaccount.
ThecompositionofTeamswithtwointernationalPeerswasgenerallyevaluatedpositively:Itseemshelpfultobeabletoexchangeviewswithanother internationalPeer,especiallywithregardtotheexternalviewofthesystem.ItisalsopossibleinthiswaythattheinternationalPeerssplitintodifferenttandemsduringthePeerVisit.Fromtheprovider'spointofview,bringinginexperiencefromtwoothereducationsystemsand/orVNFILprocessesisbeneficial.ThiscanonlybeachievediftheinternationalPeersarenotfromthesamecountryorareemployedindifferenteducationalsubsystems.
7ThishasalsobeenarecurringphenomenoninprevioustransnationalPeerReviewprojects.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 33
4. EffectsofthePeerReview50%ofVNFILprovidersansweredinthesurveythattherewereadditionaleffectsofthePeerReviewapartfromfollowinguponthefeedbackreceived.Effectsmentionedinthesurveywereimplemen-tationof special interest groups (1 answer),more involvement inpolicymaking (1), organisationallearningonqualityassuranceandnewevaluationmethods(1),impactonprofessionaldevelopmentofstaff(1)andvalidationbecomingpartoftheaimsoftheinstitution(1).
Additionally, having staff of one’s own institutionparticipate in the PeerReview(s) of otherVNFILprovidersalsohadadditionaleffectsonthatinstitution,becauseitisagreatlearningopportunityforthePeersaswell.
AsaPeeryoubringbackalotofimplicitandexplicitknowledgetoyourownorganisation.The insight into the validation procedures of other institutionsmade the viewmore clear for thepossibilitiesthatexistatone'sowninstitution.(Provider)
4.1.1 Institutional„returnoninvestment“ofthePeerReview
ThePeerReviewmethodwas judged tobebeneficial to the institution in severalways. Formanyinstitutions, this formof quality development has been a new experience and is regarded a goodcomplementtosummativequalityassuranceprocedures.Itwasseenasabenefittohavespecialistsinthe institutionwithdifferentperspectivesandvisionsandaviewfromoutside.Thestudyofthespecific institutional issues by Peers offered the examined institutions anoutsideperspective. Theinstitutionalself-imagecouldbecomparedwiththeexternalviewofthePeers.
PeerReviewallowsaverydeepinsightintothevisitedorganisationthatcannotbeachievedbymerevisitsorexchanges.TheopennessandhonestyofthePeersandthetrustingandcollegialrelationshipareparticularlyappreciated.TheroleofPeersistohelptheinstitutiondevelopfurther(andwasalsorecognisedassuch)whichenhancestheusefulnessofthePeerReviewanditsfindings.Theprocessofdealingwithconcreteproblemsof theworkhadamotivatingand inspiringeffectonallpartici-pants.Positive feedback fromthePeers isperceivedasconfirmation.Someof theresultscouldbeimplementedinaconcretewayorencouragedalonger-termdevelopment.ParticipationintheEu-ropeanprojectitselfandtheopinionoftheinternationalPeersisoftenusedinthecontextofmar-ketingandpublicrelations.Asanexampleofoptimaldissemination,itcanbementionedthat“Frau-enstiftungSteyr”receivedthe"AustrianNationalAwardforAdultEducation"fortheirparticipationinthisprojectinNovember2017.
ThePeerReviewsledtovariousfindingswithintheinstitutions.WritingtheSelf-Reportasaproviderrequiredacriticalexaminationofone'sownprocessesintherun-uptothePeerReview,sotheim-portanceofregularself-evaluationbecameevident.ThefindingsandrecommendationsofthePeerswereanimportantbasisforinstitutionaldevelopment.TheadvantageofPeerReviewisthatitexam-inesexactlythosetopicsthattheinstitutionhaschosentoanalyse.Thelearningprocesswasfurtherpromotedby takingon thedifferent rolesasaPeerandas representativeofaprovider.Thisway,onecanget toknow institutionswithinone’sownareaofprovisionvery thoroughlyand fromtheinside. Itwaspossibletocomparewhatissimilar,what isdifferentbetweeninstitutions,andtoin-corporatethisknowledgeintoone'sowninstitution.
4.1.2 RelevantchangesintermsofinstitutionaldevelopmentsincethePeerReview
During the Peer Review, the participating partner institutionsmostly discussed those areaswhereweakpointsweresuspectedand theneed for changewasalreadyevident.The focusof theques-tionswasbasedontheQualityAreas,alsowithregardtothedelineationoftheproblemareas.Basedontheissuesanalysed,thePeerReviewdidnotresultinanyoverallstructuralchangeintheinstitu-tion.ThepurposeofthePeerReviewwasspeciallytoclarifychangesconcerningcertainissuesortodevelopdifferentscenariosforchangeinmoredetail.Inthiscase,thePeerReviewReportservestounderpintheneedforchanges–alsoforthemanagementoftheinstitution.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 34
Insomeorganisations, institutionalchangeswereimmediately implementedasaresultofthePeerReview;inothercases, itwillbealong(er)processuntilchangescanactuallybeimplemented.Theexistenceofaneffectivequalitymanagementsystemwillhelptofollow-throughonchanges.
Ingeneral,PeerReviewhasapositiveeffectontheawarenessforqualityissuesandthewillingnesstoreflectone’sownwork.OneexampleforthiswastheclarificationofrolesbroughtaboutbythePeerReviewinoneinstitution.Thiswaspossiblebecausetheresponsiblepersonsinvolvedasmanycolleaguesaspossibleinthepreparation,implementationandfollow-upofthePeerReview.
Yes,we had changes and development. The Peer Reviewwas like a confirmation of the develop-ment-plans, that they are o.k. Now we have a new structure of the portfolio-procedures. A fewweeksagowehadtheofficialstartofthenewones.(Interview)
4.1.3 GoodpracticesinVNFILduringPeerReviewtransferredtootherinstitutions/countries
ThefeedbackfromprojectpartnersshowsthatPeerReviewoffersgoodconditionsforthetransfergoodpracticesofVNFILtootherinstitutionsand/orothercountries.Thebasicideaofvalidationinthe institutions is always the same, but there are differences between systems that often do notallowacompletetransfer.Ingeneral,itwillbesub-aspectsthataretransferredfromonesystemtoanother.However,itisoftenhardtoseefromwheretheinfluencesthatcausechangesintheVNFILprocess of the institutions ultimately originated. Effects are most noticeable when changes takeplaceimmediatelyafteraPeerReviewandaredocumentedinthePeerReviewReport.
Isawalotofnicemethodsandinstrumentsthatarebeingused.Theycanbetransferred.(Interview)
4.1.4 LessonslearnedduringthePeerReviewintermsofinstitutionalQAandQM
For somepartners, thePeerReviewmethodwascompletelynew.For them, the learningeffectatthe first visits as Peer, but also in the role as a providerwas the biggest. Some already knew themethodfromthenationalcontext.TheVNFILprovidersconsideredhowthePeerReviewmethodforVNFILcouldbeintegratedintothequalitymanagement.PeerReviewwasseenasagoodmethodofqualitydevelopment,complementingotherexternalevaluationsandauditsthatrathersupportqual-ityassurance.
Aboveall,thecollegialaspectwashighlightedaspositive.TheeffectisdifferentinthePeerReviewthan inanauditbecause it isnotabouthavingtoobtainacertificate. It isalsoadvantageousthat,unlike certifications, oneor two individual questions can be examined very intensively. Therefore,thePeerReviewasasupplementtoexistingQAproceduresmakessense.
Whether concrete activities for implementing Peer Review will follow depends on the extent towhichVNFILisalreadyanchoredonthepoliticalandlegallevels.ConcretediscussionswitheducationpolicystakeholdersaboutPeerReviewforqualitydevelopmentofVNFILhavealreadybeenheld intheNetherlands. IfVNFIL isnotyetanchored,PeerReviewcanhelptobuildupor furtherdevelopthesector.
Aprovidersaid,thatthemethodofPeerReviewisparticularlysuitableforbuildingupalotofknow-how in a short time,which is especially helpful at the timeof establishment of VNFIL.OnlywhenVNFILisalreadyfirmlyestablished,qualityassurancemethodssuchasauditsseemappropriate.Thedevelopment-oriented featureofVNFIL couldbeamotivation for government toprovide financialresourcesforPeerReviewforVNFIL,especiallyincountrieswhereVNFILisnotimplementedyet.ThefindingsfromVNFILthroughPeerReviewcanalsobeusedonthepoliticallevelforfurtherplanning.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 35
5. GeneralassessmentofPeerReview
5.1 ApplicabilityandaddedvalueofPeerReviewforVNFIL
PeerReviewhasprovedtobeverywellapplicableinVNFILandhasmetwithveryhighapprovalbyVNFILprovidersandPeers:
§ According to the answers of the providers’ online survey, 54.5%will certainly conduct a PeerReviewagain, theotherswill considerdoing so.Noproviderprecluded implementingPeerRe-viewinthefuture.Only27.3%,however,plantodoatransnationalPeerReview,with itsaddi-tionalchallenges,again.
§ When asked if they would recommend other VNFIL providers to use Peer Review, 81.8% an-swered„yes“,theother18.2%answered„yes,butwithreservations“,thereservationsconcern-ingtime(2outof11)andfinancialresources(1outof11).
§ (Almost)allPeersrecommendbecomingaPeertootherprofessionals,withtheexceptionofonePeer(outof28),whodidnotspecifyanyreasons.
PeerReviewandVNFILhaveparallelgoalsandprocessesandarethereforeconsideredtobea“goodmatch”:VNFILisalwaysaboutdiscoveringormakingawareofcompetenciesofaperson.Thus,thepersonupgradesherorhisqualificationsandbecomesmoreself-confident.PeerReviewhassimilargoalsandeffects:Itisaboutevaluatinguponrequestandateyelevelandtogivefeedbackinacon-structiveandappreciativeway.
ThepilotphasehasalsoshownthatPeerReviewshaveapositiveeffectonthecontinuedorganisa-tionalandprofessionaldevelopmentofVNFILprovidersandPeers.
5.2 Potentialof (transnational)PeerReviewforthedevelopmentofVNFILandqualityassuranceinVNFIL
Theaddedvalueof transnationalPeerReview forVNFIL is thatdifferences innational and institu-tionalprocessesbecomevisible.Exemplaryaspects canbeadoptedbyother systems thatare lessdevelopedintermsofVNFIL.Forexample,ifVNFILislessdevelopedinonecountryduetounfavour-ablepolicypriorities, impulses fordevelopmentcanbegleanedby lookingat thepracticeof thosecountries and/or providerswho aremore advanced. These experiences can then possibly also bepassedontothepoliticallevel.
BecausetheredoesnotexistaninternationalbasicstructureforVNFILyet,itisimportantthattrans-national Peer Review supports the development of an international standard at a higher level ofVNFILprocess:BestpracticesofallinstitutionsprovideanimprovedstandardofVNFIL.
5.3 FurtherdevelopmentofinstitutionalqualityassurancethroughPeerReview
IntheinterviewspartnersconfirmedtheirinterestincontinuingtoapplyPeerReviewintheirinstitu-tion.ThepossibilitytocombinePeerReviewwithotheralreadyestablishedqualityassurancemeth-odsisassesseddifferently.Ontheonehand,PeerReviewisflexibleenoughtointegrateitintoinsti-tutionalqualityassurance.Ontheotherhand,theadditionalexpenditureoftimeandresourcesisconsideredanobstacle.Someinstitutionsmustmeetqualitycertificationrequirements(e.g.ISOcer-tificate),forthemPeerReviewcanbeanadditionalactivitytomandatoryqualityassurance.Con-ductingaPeerReviewisespeciallyconsideredwhennewissuesanddevelopmentsneedtobetack-led.Theinternationalfeatureisexpectedtoprovideadditionalperspectivesforsolvingveryspecificquestions.Additionally,thePeerReviewisalsoseenasanopportunitytoevaluatequalitymanage-mentsystems.
Currently,ISOiscompulsoryandPeerReviewthe“freestyle”section.FromthepointofviewofapartnerPeerReviewcouldbeintegratedintoISOcertificationsothat(some)auditscanalsobere-
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 36
placedbyPeerReviews.8ThiswouldoffertheopportunitytooptforPeerReviewwhilestillcomply-ingwithISO,whichisanecessityformanyeducationaleducationswhoneedaqualitylabeltobeabletocompeteinpublictenders.
5.4 Challengesforimplementing(transnational)PeerReviewinVNFIL
There are several challenges for implementing Peer Review as a formative external evaluation forVNFILonthenationalandinternationallevel.PublicfundingforPeerReviewsisseenbytheprojectpartnersasthemostimportantrequirement.
Also the specificity of the process- and development-orientation of Peer Reviewwas emphasised.This means that the system has to be flexible enough for further development. The question iswhether there is (still) a need for this kindof evaluationprocedure ifVNFIL is (more) established,especiallywhenPeerReviewiscompetingwithother(obligatory)standardisedaudits.
Fromaninternationalperspectiveitshouldberemembered,thatlegalanchoringisnotimplementedineverycountry. Inthosecountries increasedeffortsareneededfordevelopingand implementingVNFIL.Peerreviewcouldmakeanimportantcontributiontothis–iffundingforacoordinatingstruc-tureandfortransnationalactivitiesisavailable.
8Actually,researchonPeerReviewshowedthatastrandofPeerRevieworiginallydevelopedfromspecialtypesofsecond-partyauditsor“dress-rehearsals”forthird-partyauditsinISO(cf.Gutknecht-Gmeiner2008).
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 37
6. FeedbacktoManualandToolboxandrecommendationsforfinalisation
6.1 ManualCompliancewiththeManualisthepreconditionforagoodPeerReview–thiswasalsocorroboratedbypartnersintheirfeedback.
The Manual was evaluated positively by the partners and deemed suitable for the evaluation ofVNFIL.AllPeerReviewsbyandlargefollowedtheprocedureaslaiddownintheManual.TheMeta-evaluationrevealedonlyproblemsthatwereduetodeviationsfromtheManual.
Nevertheless,whereassomesaidthatnoadjustmentsneededtobemade,someproposeda“simpli-fication”oftheManualinordertomeetthetimeandresourceconstraintsofthepeopleinvolvedinthePeerReview.Othersaskedformoredetailandmorepractical instructionsandexamples.Withtheseconflictingexpectations,itwasdecidedduringtheBratislavathatthecarefulbalanceofacon-cisebutsufficientlydetaileddescriptionofPeerReview in theManual shouldbemaintained.Part-nersarefreetoaddchecklistsandflowchartstosupportinexperiencedcolleagues.
6.2 QualityAreas
TheQualityAreasand the indicatorswereusedsuccessfullyduring thepilotphase, therewerenoindicationsofproblems.Theoverallassessmentwasthattheywerewell-suitedtostructurethePeerReviewandthattheyhelptofocusonimportantissues:
§ 79.2%ofPeersfoundtheQualityAreasandoutcomes/indicatorsforVNFILproviderseithersuit-ableorverysuitableinthecurrentformat.20.8%foundthemnotsosuitable,allofwhichmen-tionedthattheQualityAreasaretoolonganddetailedand"trytoachieveeverything,incorpo-rateeverything".SuggestionsweretoeitherdownsizetheQualityAreasortocreateanaddition-aloverview(which,infact,alreadyexists).
§ Over90%oftheparticipatingVNFILprovidersfoundtheQualityareasandoutcomes/indicatorsforVNFILprovidersinthecurrentformateithersuitable(63,6%)orverysuitable(27,3%).
ThatthestructureoftheQualityAreasfollowstheprocessofVNFILwashighlightedasaparticularlypositiveaspect.Theywereusedto reflecton thedifferentphasesof thevalidationprocessandtoidentifythoseareasthatshouldbeconsideredinthePeerReview.Thereportingofoutcomes,exam-plesofcriteriaandsourcesofevidencefacilitatedthewritingofboththeSelf-ReportandthePeerReview Report. The Quality Area “Quality Assurance” (now “QualityManagement”) wasmisinter-pretedbysomepartners.
TheQualityAreasaresocomprehensivethattheinstitutioncanpickoutaspectsofcurrentinterestandtotakeintoaccountthestageofdevelopmentofVNFILintheinstitution.ThebreadthalsoofferstheopportunitytocloselyexaminedifferentareasinsuccessivePeerReviews.
SincetheQualityAreasmustbekeptcomprehensive inorder to1)cover thewholeVNFILprocessand 2) be applicable across countries and requests for changeswere contradictive itwas decidedduringtheBratislavatokeeptheQualityAreasastheyare–withsomeimprovementstobemadeintheformulationoftheQualityAreaonQualityAssurancetopreventfurthermisinterpretations.
6.3 Toolbox
The Toolbox was considered very good and helpful. Partners appreciated that tools were flexibleenoughbutclearwhenitcomestogivingguidancefortheirwork.Twoforms(GenderMainstream-ingtool,QualityAreaAssessmentForm)werenotusedbutasrecognisedusefulforthePeerReview.EspeciallytheGenderMainstreamingChecklistwasseenasanecessityforaEuropeanproject.
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 38
Critical feedback to the toolbox largely oscillated between the forms being “too detailed” or “notdetailed enough” so that, overall, the tools seem to strike a good balance between the two ex-tremes.
NocommentsweremadeontheredundantbasicinformationintheInitialInformationSheettotheSelf-ReportandPeerReviewReportwhich isnecessaryformaintainingasatisfactory levelof infor-mation in all documents. It seems that the informationon thepossibility of copy-pasting identicalinformationfromoneformtothenextwaspassedonsuccessfullyduringtrainingandthroughdirectadvice.
Insomeinstances,partnerschangedformsthatwereonlysupportivedocumentsandnotmandato-ry.Thiswasthecaseespeciallyfortheinterviewminutes–Peersmodifiedtheformorfounddiffer-entwaysofdocumentinginterviews.ThisisinlinewiththeEuropeanPeerReviewprocedureaslongassufficientdocumentation is takingplace.Thesameholdsgoodfor themodelPeerVisitAgenda,whichcanbeadaptedorsupplantedbyownforms.TheQualityAreasAssessmentformwasnotusedatall.ItseemsthatitspurposeasaninstrumentofpreparingthecentralpartofPeerReviewReportduringthePeerVisit(withouttheadditionalchaptersandinformationaskedforinthePeerReviewReport)wasnotclear.
Partners suggested incorporating theGenderMainstreaming form into the Self-Report in order tostressitsimportancefromtheverybeginning.Thiswould,however,resultinamuchmorecompre-hensive and time-consuming report – which is contrary to the aim of making documentation as“light” as possible. Central information on gender splits (candidates/participants; staff) is alreadyincludedintheSelf-Reportform.
Themeta-evaluation toolwas ratedasenrichingsince itpromotes theexchangeamongPeersandpartners.Itwasalsodeemedveryusefulforthecoordinatorwhoneedstowritethereport.
ThedifficultiesPeersexperiencedinfillingoutthePeerReviewReporthavebeendiscussedabove.NochangesareneededsincetheproblemsdidnotlieinthereportstructurebutratherinthewaythePeerReviewwassetupand/ordeveloped.Insomeinstances,PeerTeamsorCoordinatorsalsomissedcrucialanalyticandreportwritingskills.Itis,however,recommendedtopointouttofutureusersthatthereportstructurecanbeadapted(e.g.addingageneralfeedbacksectionorasectionforrecommendations,ifnecessary)aslongasthecorepartsremain.
Inthesameveinitcouldbehelpfultopointoutmorecandidly• whichpartsofthetoolboxaremandatoryforensuringahigh-qualityandtransparentPeerRe-
view–namelytheInitialInformationSheet,theSelf-Report,thePeerReviewReportaswellasthe“GroundrulesforPeers”asafundamentalsetofrulesgoverningthePeerReviewprocessfromtheperspectiveofPeers–
• andwhichformsandchecklistsprovidehelpandguidancebutmayalsobechangedorinsomecasesalsoomittedorreplaced(allothers).
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 39
7. Documentsand(re)sources
7.1 Basicdocuments
EuropeanPeerReviewManualforVNFIL
PeerReviewToolboxforVNFIL
PeerTrainingProgramme
7.2 Documentationofpilotphaseandmonitoringdata
PeerReviewpilotdatabase:overviewofpilotPeerReviews
PeerregistercontainingallinformationfromPeerapplications
Documentation of trainings: Training programmes in English (aims, contents, agenda/schedule,methods),listofparticipants
PeerReviewdocuments(Tool-box)§ Self-reports§ PeerReviewAgenda§ Meta-evaluationofPeers§ PeerReviewReport§ OtherdocumentationofPeerReview,ifpossible:presentations,documentationoffeedback
session,interviewandobservationguidelines,interviewprotocols,photos
7.3 Reportsandpublications
Gutknecht-Gmeiner,Maria(2008):ExterneEvaluierungdurchPeerReview.Qualitätssicherungund-entwicklung inderberuflichenErstausbildung.Wiesbaden:VSVerlag. (Doctoral thesis,Uni-versityofKlagenfurt2006).
Sprlak,Tomas(2017):SynthesisReportonVNFILinpartnercountries,Bratislava.
Gutknecht-Gmeiner,Maria; Kroiss, Sophie (2018):Mentoring Report for the Project „Peer ReviewVNFILExtended“,Vienna.
Gutknecht-Gmeiner,Maria(2018):PeerTrainingProgrammefortheProject„PeerReviewVNFILEx-tended“,Vienna.
Gutknecht-Gmeiner,Maria(2018):PeerReviewLabelandQualityAssuranceforPeerReview,Vienna.
7.4 Listofinterviewpartners
BurtscherKlaudia,FrauenstiftungSteyr
Fuchs-WeiklFranz,AKSalzburg
GonçalvesSusana,Citeforma
GuimaraesTeresa,Citeforma
OsterhoutKeesvan,VigorTransitions
PaulusChristina,BOKU
SmitMarloes,Libereaux
WagnerGiselheid,WBA
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 40
7.5 Participantsreflectionworkshop
AllparticipantsoftheprojectmeetinginLisbon
ErikKaemingkMarloesSmitMadelineEichnerEvaBrazdilovaFrancescaOpertiFranzFuchs-WeiklHeidiWagnerChristinaPaulusIngaPuisaIsabelMiguelMichaelaFreimüllerSabineFischerSusanaGoncalvesTeresaGuimares
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 41
7.6 Additionaltables
Table22: PeersperPeerReview
Code VNFILproviderPeers
NationalTrans-national Total
16_01_NL LibereauxBV,TheNetherlands 3 3 6
16_02_AT Weiterbildungsakademie(wba),Austria 2 2 4
17_03_LT VilniusVocationalTrainingCentreforServiceBusinessSpecialists,Lithuania
5 2 7
17_04_FR CentreInterinstitutionneldeBilandeCompétences(CIBC),Bour-gogneSud,France
0 2 2
17_05_PT CQEPCITEFORMA,Portugal 2 2 4
17_06_ATUniversityofNaturalResourcesandLifeSciences(BOKU),Unitoflifelonglearning,Austria 2 2 4
17_07_AT FrauenstiftungSteyr,Austria 2 3 5
17_08_PT ISLASantarém,Portugal 2 3 5
17_09_AT AKSalzburg/BFI,Austria 1 3 4
17_10_NL EVCCentrumVigor,TheNetherlands 2 4 6
17_11_SK NárodnýústavceloživotnéhovzdelávaniaNÚCŽV,Slovakia 2 3 5
Totalpilotphase 23 29 52
Source:PilotDatabase
Table23: Peers:distributionbysex
Sex NºPeers %oftotal
F 18 64
M 10 36
Total 28 100
Source:PilotDatabaseandPeerApplications(merged)
Table24: Peers:distributionbycountry
Country NºPeers %
Austria(AT) 9 32%
France(FR) 1 4%
Lithuania(LT) 5 18%
TheNetherlands(NL) 6 21%
Portugal(PT) 4 14%
Slovakia(SK) 3 11%
Total 28 100%
Source:PilotDatabaseandPeerRegister(merged)
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 42
Table25: InstitutionalbackgroundofPeers
Typeoforganisation NºofPeers %
VNFILprovider 8 30%Highereducation/researchinstitu-tion 4 15%
Adulteducation 4 15%
Educationalauthority 3 11%
Socialpartners 3 11%
Non-Profit-Organisation 2 7%
Consultingfirm 1 4%
Enterprise 1 4%
Vocationaleducation 1 4%
Total 27 100%
Source: PeerRegister,N=27
Table26: TypesofPeerTrainingcompletedbyPeers
TypeofPeerTraining NºofPeers %
EuropeanPeerTraining 18 64%
NationalTraining 6 21%
OtherPeerTraining 6 21%
Notraining 3 11%Sources:PilotDatabase,DocumentationofEuropeanandNationalPeerTrainings;OnlineSurveyofPeers;
datamerged;N=28;multipleanswerspossible
Table27: NºofPeerTrainingsperPeer
NºofPeerTrainings NºofPeers %
1Training 20 71%
2Trainings 5 18%
NoTraining 3 11%
Total 28 100%
Source:PilotDatabase,documentationofTrainings,OnlineSurveyofPeersmerged,N=28
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 43
Table28: PreviousQualityAssuranceTrainingofPeersTypeofTraining NºofPeers %ISOauditor 7 27%
ISOinternalauditor 7 27%ISOexternalauditor 2 8%
EFQMassessor 2 8%EFQMinternalassessor 1 4%EFQMexternalassessor 2 8%
Other 6 23%Nonesofar 13 50%Total 26 100
Source: PeerRegister,N=26
Table29: Review/evaluationskillsofPeers(self-assessment)
Expertisein... excellent good fair basic nonesofar
Conductinginterviews 12 8 5 1 0
Conductingobservations 5 11 6 3 1
Analysingquantitativedata 6 6 11 2 1
Analysingqualitativedata 9 9 7 1 0
Givingoralfeedback 13 9 4 0 0
Writingreviewreports 8 12 4 2 0
Reviewworkinaforeignlanguage 2 10 5 4 5
Moderatinggroups 11 10 3 1 1
Conflictmanagement 7 12 4 3 0
Timemanagement 8 10 5 3 0ScientificevaluationsintheareaofVNFIL 1 8 2 4 11
Source:PeerRegister,N=26
Table30: ParticipationofPeersinPeerReviewofowninstitutionParticipationinPeerReviewofowninstitution Nº %
Yes 16 64
No 9 36
Total 25 100
Source: OnlineSurveyofPeers(N=25)
AssessmentofPilotPeerReviewsandReflectionReport
PeerReviewVNFILExtended2018 44
Table31: HowwasthecooperationwithinyourinstitutioninthepreparationandconductofthePeerReview?
CooperationinProvider NºofProviders %Verygood 7 63,6
Good 4 36,4Someconflicts -- --
Notgood -- --Total 11 100
Source:OnlineSurveyofProviders(N=11)
Table32: WasthetimeforthePeerVisitsufficient?
Sufficienttime NºofPeers %Yes 12 50
Overallyes,butsomeproblemsbecauseoftimepressure 12 50
Seriousproblemsbecauseoftimepressure -- --Total 24 100
Source: OnlineSurveyofPeers(N=24)