GO WOTRO 2011-2012Impressions from a selection jury memberby Fred Roozeboom

WISE event, Sept. 5, 2013

GO WOTRO 2011-2012Impressions from a selection jury memberby Fred Roozeboom

Who am I- 2007- present Part-time professor in Applied Physics dept. of Plasma & Materials Processing) at TU/e - 2009 - present Senior scientific advisor at TNO

- 1983-2009 Research Fellow at Philips and NXP

- 1980-1983 Scientist at Exxon-Mobil (USA)

- 1976-1980 PhD student at Univ. Twente (Chem. Engng.)

NWO’s request

~ 6 months in advance

- Phone call referring to my previous VENI jury membership, 2 years earlier, requesting to be available to review max. 10 proposals on 2-3 meeting days for the “Cross-Border” VIDI proposals, 2012 edition GO WOTRO

- Quite some insistence on the part of NWO

- Jury of 12 pre-advisors (all Dutch) with very different backgrounds

Shortly after my acceptance

- E-mails that you will get ~5 envelopes with all (~44 !) proposals

Initially.. it seemed that each member had to take only ~ 10 proposals

> 3.49 no granting

Key ranking criteria• Quality of researcher CV, h-index, international experience, experience with coaching students, independent research line

• Quality, innovative nature and scientific impact of the proposal Relevance research, well written, feasibility of the proposal

• Knowledge utilization Elaboration knowledge utilization, applicability results

Soon .. came the request sent afterwards to take and rank all proposals ..

NWO’s argument: to ensure that all members have gone through all the pieces well and can compare the sessions

GO WOTRO: a wide range of programs

A selection fromhttp://www.nwo.nl/en/about-nwo/organisation/nwo-divisions/wotro

• Food & Business Research• Agriculture beyond Food• Conflict and Cooperation over Natural Resources in Developing

Countries (CoCooN)• Global Health Policy and Health Systems• Integrated Programs• Joint MFS II Evaluations of development interventions• Migration, Development and Conflict• Urbanising Deltas of the World

I recall: Bio-medical, medical-psychological (very often) Cognitive recognition, RF-transmission

Verzonden stukken naar GO Vidi Commisie(te verwachten) verstuurdatum

Dossiernr Naam Aanvraag Adviezen Weerwoord Opmerking Referee scores1 016.128.301 Dr. H. Gommer09-Feb 09-Feb 09-Feb2 016.128.302 Dr. L.V. van de Poll-Franse09-Feb 09-Feb 09-Feb3 016.128.303 Dr. E.A.W.H. van den Hoven /TUe09-Feb 09-Feb 09-Feb zeer goed A+ A+ A4 016.128.304 Dr. ir. D.S. Draganov09-Feb 09-Feb 09-Feb Excellent A+ A+ A5 016.128.305 Dr. ir. M.W. Ertsen09-Feb 09-Feb 09-Feb6 016.128.306 Dr. J .R. van Ginkel09-Feb 09-Feb 14-Feb7 016.128.307 Dr. A. Patel09-Feb 09-Feb 14-Feb8 016.128.308 Dr. D. Navarra09-Feb 09-Feb 09-Feb9 016.128.309 Dr. ir. D.R.M. Langers09-Feb 09-Feb 09-Feb

10 016.128.310 Dr. S.A. Adams09-Feb 09-Feb 09-Feb11 016.128.311 Dr. ir. J .M. Broersen09-Feb 09-Feb 09-Feb12 016.128.312 Dr. E. Bähre09-Feb 09-Feb 10-Feb13 016.128.313 Dr. D. Seo09-Feb 09-Feb 09-Feb14 016.128.314 Dr. M. Bartels09-Feb 09-Feb 09-Feb15 016.128.315 Dr. D. Caratelli09-Feb 09-Feb 11-Feb OK A A B16 016.128.316 Dr. H.M. Kupper09-Feb 09-Feb 09-Feb A+ A+ A+17 016.128.317 Dr. A Huss09-Feb 09-Feb 09-Feb18 016.128.318 omgezet binnen NWO19 016.128.319 omgezet binnen NWO20 016.128.320 Dr. L. van der Weerd-Meulenkamp09-Feb 09-Feb 10-Feb21 016.128.321 Dr. ir. M.A. van Huis09-Feb 09-Feb 10-Feb22 016.128.322 Dr. M.T.M. Emmerich09-Feb 09-Feb 09-Feb23 016.128.323 Dr. W.M. Bergmann Tiest09-Feb 09-Feb 09-Feb Toch OK B B B24 016.128.324 Dr. Y. Yu 09-Feb 09-Feb 09-Feb OK A B A+25 016.128.325 Dr. P.P.C.W. Kemmeren09-Feb 09-Feb 09-Feb26 016.128.326 Dr. D.T. Crommelin09-Feb 09-Feb 10-Feb27 016.128.327 Dr. B. Rieger09-Feb 09-Feb 09-Feb28 016.128.328 Dr. M.M. Bakker09-Feb 09-Feb 10-Feb29 016.128.329 Dr. B.I . Salem Bernardingetrokken ingetrokken30 016.128.330 Prof. dr. A.C. Petersen09-Feb 09-Feb 09-Feb31 016.128.331 Dr. M.A. Huisman09-Feb 09-Feb 09-Feb32 016.128.332 Dr. C.F. Beckmann09-Feb 09-Feb 09-Feb A/B B A33 016.128.333 Dr. S Ando09-Feb 09-Feb 09-Feb34 016.128.334 Dr. A. Palmigiano09-Feb 09-Feb 09-Feb35 016.128.335 Dr. ir. H.H. Haisma09-Feb 09-Feb 14-Feb36 016.128.336 Dr. ir. A. Melse-Boonstra09-Feb 09-Feb 11-Feb37 016.128.337 Dr. G.A. Blab09-Feb 09-Feb 09-Feb38 016.128.338 Dr. Y. Gonzalez-Garcia09-Feb 09-Feb 10-Feb39 016.128.339 Dr. A.W. Robaingetrokken ingetrokken40 016.128.340 Dr. N Petridou09-Feb 09-Feb 09-Feb A+/A UF A B41 016.128.341 Prof. dr. ir. A.A.J .F. van den Dobbelsteen09-Feb 09-Feb 09-Feb Excellent A+ A UF42 016.128.342 Dr. Ph. Koellinger09-Feb 09-Feb 11-Feb43 016.128.343 Dr. L.J . Waldorp09-Feb 09-Feb 09-Feb OK A+ UF B44 016.128.344 Dr. H. Hung09-Feb 09-Feb 09-Feb45 016.128.345 Dr. ir. E. van Kleefingetrokken ingetrokken46 016.128.346 Dr. S. Ghosh09-Feb 09-Feb 14-Feb47 016.128.347 Dr. ir. J .F. Lichtenauer09-Feb 09-Feb 09-Feb

GO WOTRO 2011-2012 list of submitted proposals

Overall jury scores before 1st meeting in March ‘12Voorlopige prioritering totaalPositie Proj. aanvrager I . I I . I I I . I . I I . I I I . Preadv A Preadv B1 016.128.316 Dr. H.M. Kupper1.62 1.31 2.74 1.47 0.9 0.6 1.12

016.128.304 Dr. ir. D.S. Draganov1.66 1.79 2.68 1.73 0.5 0.6 0.7

Roozeboom

3 016.128.333 Dr. S Ando1.58 2.47 0.00 2.03 0.6 0.4 0.04 016.128.334 Dr. A. Palmigiano1.88 2.32 0.00 2.10 0.7 0.8 0.05 016.128.306 Dr. J .R. van Ginkel2.20 2.34 2.19 2.23 1.0 1.2 1.46 016.128.314 Dr. M. Bartels1.85 2.71 4.01 2.28 0.5 1.1 1.67 016.128.331 Dr. M.A. Huisman2.25 2.50 2.91 2.38 0.3 0.4 0.98 016.128.326 Dr. D.T. Crommelin2.62 2.27 0.00 2.45 0.7 0.7 0.09 016.128.321 Dr. ir. M.A. van Huis2.31 2.99 0.00 2.65 0.6 0.7 0.010 016.128.325 Dr. P.P.C.W. Kemmeren2.57 2.76 0.00 2.67 0.7 0.6 0.011

016.128.324 Dr. Y. Yu2.89 2.89 2.12 2.70 0.4 0.9 0.6

Roozeboom

12016.128.332

Dr. F Beckmann2.39 3.02 3.88 2.71 0.8 0.5 0.4Roozeboom

13 016.128.335 Dr. H. Haisma2.88 2.73 2.99 2.81 1.3 0.9 0.514

016.128.315 Dr. D. Caratelli2.73 3.10 4.11 2.92 0.5 0.6 0.8

Roozeboom

15 016.128.305 Dr. ir. M.W. Ertsen2.56 3.31 0.00 2.94 0.9 1.2 0.016 016.128.330 Prof. dr. A.C. Petersen2.59 3.44 0.00 3.02 0.7 0.6 0.017 016.128.302 Dr. L.V. van de Poll-Franse2.50 4.42 2.96 3.10 0.7 0.8 0.618 016.128.328 Dr. M.M. Bakker2.87 3.53 0.00 3.20 1.0 0.7 0.019 016.128.337 Dr. G.A. Blab3.28 3.12 0.00 3.20 0.5 0.5 0.020 016.128.311 Dr. ir. J .M. Broersen2.80 3.64 3.57 3.20 0.6 1.0 0.721

016.128.341Prof. dr. ir. A. van den Dobbelsteen3.04 3.66 0.00 3.35 1.2 1.2 0.0

Roozeboom

22 016.128.310 Dr. S.A. Adams3.40 3.55 4.31 3.48 0.9 0.9 1.023

016.128.303 Dr. E.A.W.H. van den Hoven3.29 4.07 4.38 3.68 1.0 1.3 1.3

Roozeboom

24 016.128.317 Dr. A Huss3.52 4.10 0.00 3.81 0.8 1.2 0.025 016.128.322 Dr. M.T.M. Emmerich3.21 4.86 4.23 3.88 0.6 1.1 0.826 016.128.336 Dr. ir. A. Melse-Boonstra3.50 4.41 4.46 3.96 0.6 0.7 0.727 016.128.309 Dr. ir. D.R.M. Langers3.95 3.98 0.00 3.97 1.0 0.9 0.028 016.128.320 Dr. L. van der Weerd-Meulenkamp3.55 4.69 0.00 4.12 1.1 0.6 0.029 016.128.313 Dr. D. Seo3.88 4.60 4.70 4.24 0.8 1.4 1.230

016.128.323 Dr. W.M. Bergmann Tiest4.50 4.30 0.00 4.40 0.8 0.9 0.0

Roozeboom

31016.128.340

Dr. N Petridou3.99 5.22 0.00 4.61 0.7 1.3 0.0Roozeboom

32 016.128.327 Dr. B. Rieger4.62 4.80 5.20 4.71 0.6 0.9 0.933 016.128.338 Y Gonzalez-Garcia4.32 5.35 0.00 4.84 1.0 1.0 0.034

016.128.343Dr. L.J . Waldorp4.76 5.00 0.00 4.88 0.7 0.8 0.0

Roozeboom

35 016.128.347 Dr. ir. J .F. Lichtenauer5.12 4.98 4.60 4.96 1.0 0.8 1.636 016.128.312 Dr. E. Bähre4.90 5.13 4.91 4.96 0.5 0.7 1.037 016.128.307 Dr. A. Patel4.61 5.37 5.39 4.99 0.6 1.2 1.338 016.128.346 Dr. S. Ghosh5.20 5.28 4.81 5.12 0.7 0.7 0.639 016.128.342 Dr. Ph. Koellinger4.73 5.74 0.00 5.24 0.8 0.8 0.040 016.128.301 Dr. H. Gommer5.05 5.49 6.07 5.27 0.9 1.2 1.041 016.128.344 Dr. H. Hung6.36 6.30 5.74 6.19 0.7 0.7 0.842 016.128.308 Dr. D. Navarra6.72 7.13 5.87 6.61 1.1 1.4 1.3

0verzicht per crit Stdev per crit

> 3.49 no granting

After case-by-case reconsideration at 1st meetingtotaal

Positie Proj. aanvrager I . I I . I I I . I . I I . I I I . Preadv A Preadv B M/V1 016.128.316 Dr. H.M. Kupper1.62 1.31 2.74 1.82 0.9 0.6 0.0 v2 016.128.304 Dr. ir. D.S. Draganov1.66 1.79 2.68 1.95 0.7 0.8 0.0 Roozeboom m3 016.128.333 Dr. S Ando1.58 2.47 0.00 2.03 0.8 0.9 0.0 m4 016.128.334 Dr. A. Palmigiano1.88 2.32 0.00 2.10 0.8 1.2 0.0 v5 016.128.306 Dr. J .R. van Ginkel2.20 2.34 2.19 2.23 0.5 1.1 1.6 m6 016.128.331 Dr. M.A. Huisman2.25 2.50 2.91 2.48 0.6 0.7 0.0 m7 016.128.326 Dr. D.T. Crommelin2.57 2.39 0.00 2.48 0.8 1.1 0.0 m8 016.128.314 Dr. M. Bartels1.85 2.71 4.01 2.61 0.8 0.9 0.8 v9 016.128.321 Dr. ir. M.A. van Huis2.31 2.99 0.00 2.65 0.5 0.4 0.0 m10 016.128.325 Dr. P.P.C.W. Kemmeren2.57 2.76 0.00 2.67 1.0 1.5 1.4 m11 016.128.324 Dr. Y. Yu3.24 2.94 2.12 2.89 0.9 1.1 1.0 Roozeboom m12 016.128.335 Dr. H. Haisma3.15 2.49 3.10 2.97 0.6 0.9 0.9 v13 016.128.337 Dr. G.A. Blab3.08 2.88 0.00 2.98 0.7 0.8 0.0 m14 016.128.332 Dr. F Beckmann2.60 3.00 3.85 3.01 0.8 1.4 1.1 Roozeboom m15 016.128.302 Dr. L.V. van de Poll-Franse2.45 4.42 2.96 3.07 0.5 0.6 0.7 v16 016.128.330 Prof. dr. A.C. Petersen2.78 3.45 0.00 3.12 0.9 0.9 0.0 m17 016.128.311 Dr. ir. J .M. Broersen2.90 3.42 3.57 3.20 0.5 0.9 0.6 m18 016.128.328 Dr. M.M. Bakker3.02 3.78 0.00 3.40 0.6 0.7 0.7 v19 016.128.305 Dr. ir. M.W. Ertsen3.35 3.50 0.00 3.43 1.0 1.2 1.4 m20 016.128.303 Dr. E.A.W.H. van den Hoven3.04 3.58 4.23 3.47 0.6 0.4 0.0 Roozeboom v21 016.128.315 Dr. D. Caratelli3.28 3.68 4.11 3.59 0.7 1.0 0.0 Roozeboom m22 016.128.310 Dr. S.A. Adams3.40 3.70 4.31 3.70 0.7 0.6 0.0 v23 016.128.340 Dr. N Petridou3.40 4.12 0.00 3.76 0.5 0.8 0.9 Roozeboom v24 016.128.320 Dr. L. van der Weerd-Meulenkamp2.95 4.59 0.00 3.77 1.1 1.0 0.0 v25 016.128.317 Dr. A Huss3.52 4.20 0.00 3.86 0.7 0.6 0.6 v26 016.128.322 Dr. M.T.M. Emmerich3.21 4.86 4.23 3.88 0.5 1.1 0.7 m27 016.128.341 Prof. dr. ir. A. van den Dobbelsteen3.69 4.14 0.00 3.92 0.6 1.2 1.3 Roozeboom m28 016.128.309 Dr. ir. D.R.M. Langers3.90 3.98 0.00 3.94 0.6 0.7 0.0 m29 016.128.336 Dr. ir. A. Melse-Boonstra3.50 4.41 4.46 3.97 0.9 1.1 0.0 v30 016.128.313 Dr. D. Seo3.88 4.60 4.76 4.28 1.4 0.8 0.8 v31 016.128.323 Dr. W.M. Bergmann Tiest4.50 4.30 0.00 4.40 1.2 1.0 0.0 Roozeboom m32 016.128.338 Y Gonzalez-Garcia4.22 5.06 0.00 4.64 1.0 0.8 1.6 v33 016.128.312 Dr. E. Bähre4.76 4.88 4.71 4.78 0.7 0.5 0.7 m34 016.128.327 Dr. B. Rieger4.62 4.80 5.20 4.81 0.6 1.1 0.8 m35 016.128.343 Dr. L.J . Waldorp4.76 5.00 0.00 4.88 0.8 0.8 0.0 Roozeboom m36 016.128.347 Dr. ir. J .F. Lichtenauer5.12 4.98 4.60 4.96 1.1 1.4 1.3 m37 016.128.307 Dr. A. Patel4.61 5.37 5.39 5.00 0.3 0.4 0.9 m38 016.128.346 Dr. S. Ghosh5.20 5.28 4.81 5.12 0.7 0.7 0.8 v39 016.128.342 Dr. Ph. Koellinger4.73 5.74 0.00 5.24 0.7 0.7 0.6 m40 016.128.301 Dr. H. Gommer5.05 5.49 6.07 5.42 0.9 0.6 1.1 m41 016.128.344 Dr. H. Hung6.36 6.30 5.74 6.19 0.9 1.2 1.0 v42 016.128.308 Dr. D. Navarra6.72 7.13 5.87 6.61 0.7 0.8 0.0 m

0verzicht per crit Stdev per crit

> 3.49 no granting

Done deal for interview part ?Good to be a woman ?

Mostly: yes…, but an interview can make or break a proposal !

NABESPREKINGCommissielid ….. merkt op dat de kandidaat op een hoge positie stond voor het interview.

De voorzitter prof. ….. merkt op dat de referenten 3 x A+ hebben gegeven, maar wel inhoudelijke vragen hadden.

Commissielid …..merkt op dat de kandidaat erg zenuwachtig was.

De voorzitter prof. ….. merkt op dat er niet voldoende onderbouwing is voor type D en de oorzaken van type D en dat het goed is dat commissielid ….. de genetische studie had uitgezocht.

Commissielid ….. merkt op dat het genetische gedeelte niet haalbaar is.

De voorzitter prof. ….. merkt op dat de kandidaat vastliep in haar antwoord op de vraag over causaliteit. Dit is geen mechanistisch onderzoek.

One triple A+-case on medical-psychological as a painful exception:

2 months after interviews, when you think it’s done…An unexpected appeal letter sent to NWO, requesting a revision

• Very educational for assessing jury members and the assessed candidates

• The number of proposals for the interview selection, 45 pieces, is (too) much It's a lot of work to give all applications in the selection interviews a score. Logical that this method ensures that all members have gone through all paper work.

• The interview has a major impact on the overall assessment.

• Not all the proposals had a clear cross-border nature. How to filter out these requests?

• The selection of referees has a large influence on the entire assessment process.

• An expert was not always present in the committee for each relevant application area. Applicants should be trained in presenting for a wide committee.

• Remarkably, the utilization of knowledge was not a mandatory section in the proposal / process. This makes a good comparison impossible. It is advisable that NWO better makes the utilization of knowledge a mandatory part included in the application, or leaves it out.

• The assessment of the CVs of candidates is complicated. Not in all areas, impact factors are not universally used, making comparison of CVs complicated. It is not clear what position journals hold in different areas. Guidelines for the assessment of the CV can be submitted more explicitly to the referees; an example may be the form of the ERC.

Some other notes on the GO Vidi procedure:

The diversity of topics is inherent to the cross-border nature of this VIDI call

Too specialized proposals were referred to other domains

The remaining proposals were often bio-medical, medical-psychological, etc., of nature, and (too?) many jury members had just such a background

Very instructive for me as a jury member

Sometimes random factors cannot be completely avoided, as the proposals are first reviewed by referent-experts and provided with a score before the plenary review by the pre-advisors committee

In retrospect: the cognitive recognition candidate (with A+, A+, A) had made a minitious study on all interview committee members in an attempt what to expect, how to approach them during the interview

My own impressions and final remarks

(her interview )