an expert evaluator’s perspective · 2017-12-18 · limitless potential | limitless opportunities...

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACTLIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACTCopyright University of Reading

AN EXPERT EVALUATOR’S PERSPECTIVE

Prof. James Ferryman

1

SMPCS / Dept. Computer Science

LIMITLESS POTENTIAL | LIMITLESS OPPORTUNITIES | LIMITLESS IMPACT

OVERVIEW• What is meant by EU ‘expert evaluator’?

• How to become involved in evaluation process

• Evaluation process

• Common features of successful (and unsuccessful) proposals

• Benefits of being an expert evaluator

• Personal experience

2


WHAT IS MEANT BY ‘EXPERT EVALUATOR’?• EC: “You have a chance of being selected as an expert if you have a

high level of expertise in relevant fields of research and innovation”• Must also be available for occasional, short-term assignments, good

knowledge of English, and be able to use IT tools

• Expert assignments involve:• Evaluation of proposals

• In response to published calls (topics)• Monitoring of actions

• E.g. periodic review of ongoing projects

• Experts are offered a contract that defines the rights, obligations, terms and conditions, and they are often remunerated for their services.

3


HOW TO BECOME INVOLVED IN EVALUATION PROCESS• Register via Participant Portal

• https://ec.europa.eu/research/participants/portal/desktop/en/experts/index.html

• Expert creates his/her ECAS login and uploads biodata + CV

• Expert selection includes expertise, nationality balance, >=40% female

• Experts are independent and external, and must abide by code of conduct

• Need to sign a declaration of no conflict of interest• Maintain strict confidentiality throughout whole evaluation process

4

https://ec.europa.eu/research/participants/portal/desktop/en/experts/index.html


5

Registration for new experts

Update your existing profile


EVALUATION PROCESS

6

Receipt of proposals

Individual evaluation

Consensusgroup Panel Review Finalisation

Evaluators

Individual EvaluationReports

(Usuallydone

remotely)

ConsensusReport

(May be doneremotely)

Panel report

EvaluationSummary Report

Panel ranked list

Eligibility check

Allocation ofproposals toevaluators

Final ranked list


EVALUATION PROCESS• Remote

• At home/University, usually few days+• Evaluate all allocated proposals

• 3-5 experts assigned to each proposal• Submit IER electronically

• Central• Consensus meetings (1-3 hours) in e.g. Brussels• Act as expert or rapporteur (w/ PO as moderator)

• Panel Meeting• At end of central week• Expert may be nominated as panel rapporteur

7


GOOD EXPERTACCORDING TO REA:• “Timely in delivering reports and during consensus meetings• In case of problem informs ON TIME• Reliable• Flexible• Able and willing to learn and accept our process• Written and spoken English of good quality• Able to judge and evaluate the project, not to copy/paste• If comments are negative – WHY?• If comments are positive – WHY?

• We do not look for expert-superman, everybody can learn it...”8


COMMON FEATURES OF SUCCESSFUL PROPOSALSIs eligible (check eligibility conditions for certain topics)

Has properly answered to the call topic (is in scope)

Clearly answered: WHY? is the proposal relevant/context; WHAT? exactly is the idea/solution; WHO? will do the work (credibility); HOW? will it be done.

Contains: Clear and measurable objectives; excellent understanding of subject matter (background/context); sound concept and ambition; accurate knowledge of state-of-the-art and innovation potential; well defined expected impact; appropriate management structure; well defined dissemination and communication mechanisms; well defined exploitation plan.

Easy to read – especially first 10-20 pages

Is consistent and proof read. 9


COMMON FEATURES OF UNSUCCESSFUL PROPOSALS

• Does not properly answer to the call topic (scope)• Concept is not sound• Objectives are not well defined and/or measurable• Lacks innovation potential / TRL advance• Expected impact not well described (ref. call text)• Lacks appropriate involvement of end users / practitioners• Lacks methodological detail (e.g. in WP task descriptions)• Inappropriate management structure

• Knowledge/IPR management and/or innovation management insufficiently addressed

• Lacks partner expertise in required area• Does not convey all required details in 50pp/70pp (for 1-stage proposals)

10


BENEFITS OF BEING AN EVALUATOR• Learn differential quality of proposals -> (structure, content) that

make proposals successful• Understand common errors • Appreciate state of the art in subject area, innovations beyond• Professional exchange with other evaluators• Connect with EC / Project Officers• Helps with proposal writing e.g. innovation management, risk

management, expected impactw/ access to large number of proposals and draft ESRs

• Pride and reward in helping to make funding decisions worth € € €

11


PERSONAL EXPERIENCE• Evaluated a significant number of proposals across different priorities• Built up trust and relationship with EC evaluation units• Acted as expert, consensus meeting rapporteur, super rapporteur,

panel rapporteur • Learnt a great deal (of respect) about evaluation process

• Feedback to EC helps improves the process• Learnt a great deal on what makes for an excellent proposal

• Has helped me write significantly better proposals within a highly competitive environment

• Also evaluated ongoing projects at periodic review meetings• Working as a group with 2 other experts + PO• Gained significant experience in generating / lead on review reports

12

an expert evaluator’s perspective · 2017-12-18 · limitless potential | limitless opportunities...

Documents