university of southampton professor helen simons degeval ......creative evaluation professor helen...

Towards Professionalization of Evaluation: Voluntary Evaluator Peer Review: Is this the Road to Travel?

Zurich 11th September 2014EvaluationConference

DeGEVAL/SEVAL

Professor Helen SimonsUniversity of Southampton

© Helen Simons 2014

Structure

Hallmarks of a profession – are we there? Competencies and Capabilities Scope of UKES Capabilities framework Voluntary Evaluator Peer Review (VEPR) Eval/Partners Initiative: survey & workshop Issues arising Reflections on which road to take.


Hallmarks of a Profession 1

Purpose -public service to societySpecific body of knowledge Agreed activities as part of professionProfessional society - to further aims

and sanction bad practiceGuidelines, standards and ethical code

to uphold practice in the field


Hallmarks of a Profession 2

Training to perform agreed activities -probationary period

Continuous professional development Registration to practice & potential to

deregisterEvaluation competence/ capability


Observations on Progress Towards A Profession

Public service to society Specific body of knowledge Agreed activities Professional society Ethical code and principles Training Probationary period Registration to practice Potential to deregister if incompetence proven.


Rise of Competency/Capability Frameworks

On The Road to Voluntary Evaluator Peer Review


Reasons for Rise of Competency Frameworks

Zeitgeist of activity in evaluation societies and other organizations

Professional – extension of ethical guidelines standards; help recent evaluators develop capacity

Political - reaction to dominance of accountability; greater awareness of use and misuse

.


Competencies and Capabilities -Purposes

Purposes vary: Professional development Evaluation Training Selection Professional designation Mentoring But all designed to enhance professionalism of

evaluation


UKES Evaluation Capabilities Framework

Early issues Flexible and aspirational Not for certification at this time Track record as important a

skills/knowledge Response cautious Focused on risks of use



Later Issues –related to purpose Educative Quality assurance for ourselves Reassurance to funders and audiences of

quality of evaluators and evaluation Overall aim to promote professionalization

of evaluation as a social practice. Preference for Capability


Why Capability

Broader conception – agency Much evaluation takes place in teamsSocio/political circumstances impactResponsibility not entirely with

individual Capabilities to conduct a quality and

credible evaluation


Contexts of Use

Individuals - professional development Commissioners - skill mix for teams Organizations - build understanding

- capacity for self-evaluation - ensure quality

Framework for designing training Quality assurance process for evaluators,

stakeholders & audiences Voluntary Evaluator Peer Review



Evaluation Knowledge

Professional Practice

Qualities and Dispositions



1. Evaluation Knowledge 1.1 Understands the social and political role of

evaluation

1.2 Familiarity with evaluation designs and approaches

1.2.3 Identifies relevant evaluation questions

1.3 Comprehends & makes effective use of evaluation methodologies



2 Professional Practice 2.1 Demonstrates ability to manage and

deliver evaluations 2.1.3 Identifies data gathering instruments

appropriate to the task

2.2 Demonstrates Interpersonal skills 2.22 Shows ethical sensitivity in specific

socio/political contexts 2.26 Manages conflicts of interest & values fairly



3. Qualities and dispositions

3.2 Exercises sound, rigorous and fair judgment

3.4 Displays independence of mind ad integrity especially when evaluation challenged

3.5 Upholds democratic values in conducting & reporting evaluations


Voluntary Evaluator Peer Review



Different forms –paper/pencil, interview, professional conversation, online

CES - first - particular form Eval Partners Initiative UKES/ EES 2013/4 Two phases: survey and workshop Built on VEPR system from NZ Issues in designing a VEPR system



Eval /Partners Initiative UKES/ EES 2013/4 Concept summary:

Members apply to undergo a structured professional practice review with two accredited peer reviewers

Applicant identifies practice areas for building their professional capacity


Voluntary Evaluator Peer Review -Process

Set of questions for review related to capabilities framework

Review face-to face or via Skype or tele-conference

Reviewers recommend 1) list on VEPR index or 2) follow-up review

Lapses after 3 years unless renewed


Voluntary Evaluator Peer Review - Issues and Choices


Voluntary Evaluator Peer Review -Issues and Choices

Purpose Quality Motivation Context Whose role Voluntary or Mandatory



Quality and Integrity of Process Levels Focus Choice of reviewers Criteria for selection of reviewers Timescale



PaymentPrematerial InfrastructureGovernance


Risks (1) Costs could exclude some evaluators

Lack of consistency across reviewers

Too bureaucratic for too little reward

May result in lowest common denominator

Danger of knowledge about evaluator affecting subsequent proposal chances

Danger of society seen to be endorsing evaluator as competent whereas only signifies been through pd process


Risks (2)

Not used sufficiently to improve quality

Insufficient reviewers

Process takes too long

Disputes may take time to resolve

Clients could favour ‘approved evaluators’

Difficult of actually assessing evaluators’ capabilities beyond knowledge and skills


Voluntary Evaluator Peer Review: Is this the road to travel ?


Voluntary Evaluator Peer Review : Is this the Road to Travel ?

Benefits identified by participants Improve profile of discipline – give it a

professional stamp of approval Promote agreed standards for moving

towards accreditation and professionalization

Improve quality of evaluation



A cautionary note‘But in order to make the process work for evaluators it must be stressed that the gestation process requires extensive volunteer time, inclusive consultation that is not rushed and a willingness to ensure that the tested competencies are meaningful and adapted to the context (Mc Guire, 2012)



And a caution from me.

Quality of evaluation, as distinct from designation and/or regulation ?

It is here - in the field- that the credibility of our works stands or falls, whatever we claim and whatever credentials we hold.



End for Now

And... good luck if you follow this road

[email protected]


REFERENCES Stake, R.E. & Schwandt, T.A. ( 2006) ‘ On Discerning Quality in Evaluation’,

in Shaw, I.F., Greene, J.C. & Mark, M.M. (2006) The Sage Handbook of Evaluation, London: Sage, pp 404-418

Picciotto, R. (2011) The logic of evaluation professionalism, Evaluation, 17(2) 165-80, Sage Publications

Simons, H (2006) in Shaw. I., Greene, J.C., & Mark, M (eds.) The Sage Handbook of Evaluation, London Sage, pp243-265 for an overview of the evolution of the move towards professional ethics, their foci and purposes


Early Considerations (2009)

Flexible and aspirational

Starting point for commissioners/organizations to think about team skills required

Track record of doing evaluations as important as skills and knowledge

Guidelines for developing evaluation training Not for certification at present- needs infra

structure


Choice in selection of reviewers

Nearly a third said yes – to assure that reviewers understood their area of interest /expertise

Half said no (except for right of appeal) to ‘ensure objectivity’

Others said a voice but not total selection


Role of Evaluation Societies

Majority indicated society should take lead

External agencies less qualified to judge

Possibly work with other groups/societies

Too soon, need to incorporate existing courses

Depends on cost & capacity for setting up

Danger of lowest common denominator


Should applicants have to pay?

Majority said yes but payment figure varied

Sliding scale suggested according to workplace & expertise, commensurate with cost of reviewers & admin.

Needed payment to give the process value

Maybe reduction in membership fee to encourage take-up


Should peer reviewers be remunerated?

Over half agreed, but modestly

Somewhat dependent upon position & expertise of reviewers

Reason – doubtful that evaluators would take on task for no payment, need to attract calibre of reviewers

Minority said no; others dependent upon precise nature of scheme


Administrative Infrastructure

Keep light and demand led

Panel of 20-30 reviewers; Part/full-time admin.

Procedures for recording results, handling submissions, recruiting reviewers, criteria, timeframe, dispute mechanism, marketing

Precise form dependent upon demand and society resources - needs to be self sustainable

Enormity of task if taken up widely - keeping track, review etc.,


UKES Survey : Response

Small survey (10 questions) low response (after 3 extensions) related to concept summary

Majority responses, practicing evaluators from range of discipline backgrounds

Place of work, consultancy and public sector

View with caution but useful issues


Voluntary Evaluator Peer Review -Process

Clear statement of process & advocacy to gain support

Society committee and infrastructure Appointment of co-ordinator Selection of reviewers Applicants apply & reviewers agreed Co-ordinator liaises review time & place Applicant submits form nominating

areas & two recent projects


Potential Risks (2009)

Could become too complicated

Instrument of bureaucratic control – ‘tick box’, ‘too prescriptive’

Fail to promote diversity of evaluation contexts

Stifle innovative methods and practice


Issues from Workshops (2010) Capabilities preferable term to

competencies

Purpose should be educative - to build capacity

Competencies should relate to evaluation task rather than evaluators

Balance between technical and practical


Issues from Workshops (2010)

How to represent complexity of ‘lived experience’

Take account of dynamics of change in fluid policy environment

How to ensure does not exclude certain groups e.g. self- evaluation

How to ensure flexibility of use


Purpose of UKES Capabilities Framework

Professionalization of evaluation as a social practice

Quality assurance for ourselves

Reassurance to funders and audiences of quality of evaluators and evaluation


Further Potential Uses

Continuous professional development- Matrix of opportunities Mentoring Selection Designation of individual evaluators Register of team competencies –

procurement processes Accreditation of evaluation training


Pros and Cons of VEPR

However...

cumbersome nature of system

how to accommodate different specialismsapproaches and wider capabilities

how to get permission to share evaluation conducted in team settings

consistency across reviewers


Pros and Cons of VEPR

Majority said would endorse a VEPR system as first step in designation

‘potential to facilitate skills and knowledge transfer between peers & thus expand competencies, qualities & capacity of sector’

Should be voluntary – better able to respond to needs than if imposed

Good learning process for individuals


Should applicants have to pay?

Majority said yes but payment figure varied

Sliding scale suggested according to workplace & expertise, commensurate with cost of reviewers & admin.

Needed payment to give the process value

Maybe reduction in membership fee to encourage take-up


Reflections

…. I would hate to see any such set of competencies becoming used as a crude checklist by commissioners… to rule evaluators/trainers in or out of potential funding opportunities. I would prefer to see the society offer guidance to key competencies rather than develop an orthodoxy of ‘must haves’

( UKES Teacher/researcher/other)


Reflections

The long term goal should be to engage evaluation buyers sufficiently that they start to use commitment to the framework as a discriminating factor in placing contracts.

(UKES, Consultancy)


Reflections

A competency framework is useful in terms of professional development and training and…for those new to evaluation who need to commission work. However it should be seen as an indicative list of some of the most important competencies – it should not be interpreted too rigidly.

( Non UKES, Analyst/ Manager Public sector)


Benefits Improve profile of discipline –’professional

stamp of approval’

Encourage practitioners/ commissioners to think about capabilities for evaluation

Assuring commissioners/audiences

Providing assurance to both sides in contracting for external evaluation

Agreed standards for moving towards accreditation & professionalization

university of southampton professor helen simons degeval ......creative evaluation professor helen...

Documents