how are research proposals to sshrc evaluted

How are research proposals to SSHRC evaluated?

A look inside the black box of peer-adjudicated social science

Charles DavisRTA/FCAD

Ryerson University15 September 2005

Member, SSHRC committee 21, 2001-2004Chair, 2002-2004

Basic program features• A Standard Research Grant (SRG) is

intended to fund a 3-year research program• Up to $250K over 3 years to individual or

team– Maximum $100k/year

• 2447 SRG proposal adjudicated– 40.1% funded– 28.9% of requested funds approved (~ $80M)

• The success rate of new scholars is about 10% lower than that of established scholars

• Classics, ancient and mediaeval studies, religious studies, classical archaeology01• History: history of science, technology and medicine02• Fine arts: history and philosophy of art, architecture, theatre, music, film, dance03• Linguistics, applied linguistics and translation05• Economics07• Sociology and demography08• Geography, urban planning and environmental studies09• Psychology10• Education 1: Arts education, bilingual education, civic education, computer assisted instruction, counselling

and career guidance, early childhood, educational psychology, environmental education, geography, health sciences education, history, mathematics, moral, values and religious education, pedagogy, physical education, reading and writing, science, second language, special education and vocational education (For additional disciplines, see Committee 17) 12

• Interdisciplinary and multidisciplinary studies15• Anthropology and archaeology16• Education 2: library and information science and archival science: adult, continuing and community

education; comparative education, curriculum, distance education; educational administration, planning, and governance; history, philosophy & theory of education; higher education, measurement and evaluation, sociology of education, teacher education (For additional disciplines, see Committee 12)17

• Literature 1: English (from the Mediaeval to the Victorian period), French; German; Slavic18• Literature 2: American, modern and contemporary literatures in English, English Canadian, First Nations,

French Canadian & Québec, romance, other languages & literatures19• Health studies and social work 20• Human resources management, information systems, international business, management; marketing,

organizational studies; business policy, industrial relations21• Accounting, finance, management science, productions and operations management22• Law, socio-legal studies and criminology23• Political science and public administration24• Philosophy25• Communication, cultural studies and women's studies26

21 adjudication committees (2004-5)

golden rules of peer-adjudicated grantmaking

•The process is objective. It does not matter whether you have friends or colleagues on selection committees.•The process is not random. It is not a form of lottery.•Winning proposals are not selected.

– weaker proposals are eliminated from the competition – the winners are those that remain.

The ‘rules of the game’ perpetuate the Matthew

Effect

“Unto he that hath shall be given.

From he that hath not shall be taken away”

i.e. the funding mechanism obeys a law of accumulated advantage

Scoring formula

• Regular scholar– Record of achievement 60%– Research Program 40%

• New scholar– Whichever is higher:

• Record of achievement 60%, research program 40%

• Record of achievement 40%, research program 60%

Research achievement• evaluation of the record of research

achievement is based primarily on contributions to research the applicant has made within the last six years

• if the applicant's research career has been interrupted, research achievement is evaluated based on his or her most recent period of research activity.

• For regular scholars, applicant's five most significant contributions are taken into account in order to accurately situate the most recent six years in the context of the applicant's overall career.

Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001

Research achievement

• refereed publications, including books, chapters of books and articles;

• book reviews by the applicant/co-applicant or published reviews of his/her work;

• research reports, papers presented at scholarly meetings or conferences, and other forms of written scholarly expression or participation in public discourse and debate which constitute a contribution to research;

• where appropriate, contributions to the training of future researchers, including the supervision of graduate theses and/or the involvement of students in research activities;

• research results from previous research grants, other awards from SSHRC or other sources;

• academic awards and distinctions-new scholars may include scholarships and fellowships

Research contributions include:



• quality and significance of published work (taking into consideration the quality of the chosen publication venues);

• originality of previous research and its impact on the discipline or field;

• quantity of research activity relative to the stage of the applicant's career;

• demonstrated importance of other scholarly activities and contributions;

• recentness of output (taking into account the nature of the applicant's career pattern and previous non-research responsibilities);

Evaluation criteria:



• importance and relevance of dissemination of research results to non-academic audiences (as appropriate);

• significance of any previous research supported by SSHRC or any other agency;

• where applicable, contribution to the training of future researchers. (The committee must make allowances for applicants who have not supervised graduate students simply because their university does not offer graduate programs.)

• efforts made, where appropriate, to develop research partnerships with civil society organizations and government departments.

Evaluation criteria:


Record of research achievement

Research program: one or more projects over 3 years• explicit objectives, situated within the context

of current scholarly literature;• relationship of the proposed research to the

individual's ongoing research or to insights gained from earlier achievements-,

• importance, originality and anticipated contribution of the proposed research;

• theoretical approach or framework;• research strategies or methodologies (detailed

methodology not necessary);


Research program• plans for the communication of

research results within and beyond the academic community

• specific roles and responsibilities of students and research assistants, including how their duties will complement their academic training;

• relationship of requested budget to proposed program of research.


Research program: evaluation criteria

• degree of originality and nature of expected contribution to the advancement of knowledge

• scholarly and intellectual as well as social and cultural significance of the research

• appropriateness of the theoretical approach or framework;

• appropriateness and expected effectiveness of the research strategies or methodologies

• suitability and expected effectiveness of plans to communicate research results both within and, as appropriate, beyond the academic community


Program of research

Score needed for funding

rejected

funded

Meritorious but not funded

cutoff zone: currently about 7.3 for SRGs


5

5.5

6

6.5

7

7.5

8

8.5

9

5 6 7 8 9

research program

tota

l sco

re

achievement 6

achievement 7

achievement 8

achievement 9

rejected

Meritorious but not funded

funded


Achievement vs. research program

Some common errors

• Theoretical framework weak or insufficiently explained

• Methodology weak or insufficiently explained• A project is extended over 3 years to make it

look like a program• Budget is padded or poorly formulated• Padding of CV• “me too” proposals

– SSHRC funded research like this last year– Another research project in already worked-over

area

• Implausible teams

Some common errors

• Failure to respect page limits (6 pages means 5.75-6 pages – 6.1 pages is no good)

• Include literature review or information compilation as research

• Grad students’ roles not consistent with research program flow and objectives

• In a resubmission, failure to take into account the views of the committee and the external assessors

• Ultra cartesian or ultra baconian research designs

Risky storylines• “I’m Too Important to Submit a

Fully Worked-out Research Proposal – my record speaks for itself”– Variant: “We’re a team of Very High

Profile Researchers. Our collective Research Achievement is off your scale”

• “The fate of the world hangs on the outcome of my project”

Risky storylines• “My colleague got a grant last year to work on

hamsters, so I will work on hamsters also”• “It would please God if this proposal were funded”• “My research results will overturn all established

theories”• “The Minister mentioned that this would make a

great research project”• “Because of the proliferation of incommensurable

discourses in late postmodernity, you cannot understand what I am saying and I cannot understand my respondents, but I will study them anyway if you pay for it”

A typical 3-year program

• Year 0: literature review completed; methods and instruments selected; preliminary hypotheses formulated

• Year 1: refine instruments and hypotheses through qualitative research (focus groups, grounded theory, etc.). Test instruments

• Year 2: apply instruments for data gathering

• Year 3: analysis, interpretation, modeling, dissemination of results

Common winning formulas for new scholars

• New scholar with good track record extends doctoral research via 3-year program– Watch out. If the proposed research

is too close to the doctoral research, it will be regarded as derivative. If it is too far away, it will be regarded as too bold.

– The most successful ones have published several articles (often with their PhD supervisor) before applying for a first grant

Common winning formulas for new scholars

• New scholar as PI with established scholar as co-investigator with specified roles– The co-investigator brings up the

“research achievement” score in proportion to his/her role in the project

Budgeting tricks and traps

• the committee may reduce your requested budget.

• It is good to ask for money for grad student stipends – build grad students into your program– Note: It is best to use doctoral

students. In regional universities it is OK to use master’s students. If you use undergrads, make sure you have a good reason.

Budgeting tricks and traps• Do NOT request conference travel money

in Year 1.– Hint: OK to request modest funds for grad

student travel to conferences, if they present.

• Do NOT inflate travel cost estimates.– it is permissible to include travel costs of

work with research collaborators, but not collaborators’ research costs

• Note that research travel costs include dissemination costs, which are also calculated separately

Budgeting tricks and traps• Do NOT request funds for

computers unless computers are clearly necessary for the research and they are unavailable through the university– OK to ask for laptops for field

research

• NEVER ask for funding for less than three years

Budgeting tricks and traps• Research Time Stipends are only available if

the home university contributes one-to-one matched funding

• Do NOT request funds for books. SSHRC does not like to pay for books.

• Be CAREFUL if you request funds for consultancies – this is thin ice

• NOTE THAT once the budget is approved, SSHRC says that you can spend your grant however you like – but your University controllers do not necessarily know this. – At any rate, you cannot pay yourself an

honorarium.

Budgeting tricks and traps• See SSHRC’s list of ineligible items.

It includes: training, purchase or rental of standard office equipment, preparation of teaching materials, entertainment and hospitality costs, research leading to a degree, fees and honoraria to colleagues, indirect or overhead costs, etc.

• ALWAYS include a clear explanatory budgetary note

year 1 year 2 year 3

Master's 12000 12000 12000

RTS 3800 3800 3800

travel-applicantCanadian 1000 2000 3000foreign 2000 2000

travel-studentCanadian 2000 1500foreign

other expensesprofessional 10,000supplies 2000 1000 1000equipment 4000other equipment 1000other 1000 1000 1000

total 26800 31800 24300

Budget for hypothetical three-year, one-person small project at a small or medium (non-doctoral) university

Thank you!

Questions or comments?

how are research proposals to sshrc evaluted

Documents

research proposals

record of research achievement

environmental education

arts education

sociology of education

religious education

year research program

applicants research