Published on February 19, 2014
How are research proposals to SSHRC evaluated? A look inside the black box of peer-adjudicated social science Charles Davis RTA/FCAD Ryerson University 15 September 2005 Member, SSHRC committee 21, 2001-2004 Chair, 2002-2004
Basic program features • A Standard Research Grant (SRG) is intended to fund a 3-year research program • Up to $250K over 3 years to individual or team – Maximum $100k/year • 2447 SRG proposal adjudicated – 40.1% funded – 28.9% of requested funds approved (~ $80M) • The success rate of new scholars is about 10% lower than that of established scholars
21 adjudication committees (2004-5) • • • • • • • • • • • • • • • • • • • • • Classics, ancient and mediaeval studies, religious studies, classical archaeology01 History: history of science, technology and medicine02 Fine arts: history and philosophy of art, architecture, theatre, music, film, dance03 Linguistics, applied linguistics and translation05 Economics07 Sociology and demography08 Geography, urban planning and environmental studies09 Psychology10 Education 1: Arts education, bilingual education, civic education, computer assisted instruction, counselling and career guidance, early childhood, educational psychology, environmental education, geography, health sciences education, history, mathematics, moral, values and religious education, pedagogy, physical education, reading and writing, science, second language, special education and vocational education (For additional disciplines, see Committee 17) 12 Interdisciplinary and multidisciplinary studies15 Anthropology and archaeology16 Education 2: library and information science and archival science: adult, continuing and community education; comparative education, curriculum, distance education; educational administration, planning, and governance; history, philosophy & theory of education; higher education, measurement and evaluation, sociology of education, teacher education (For additional disciplines, see Committee 12)17 Literature 1: English (from the Mediaeval to the Victorian period), French; German; Slavic18 Literature 2: American, modern and contemporary literatures in English, English Canadian, First Nations, French Canadian & Québec, romance, other languages & literatures19 Health studies and social work 20 Human resources management, information systems, international business, management; marketing, organizational studies; business policy, industrial relations21 Accounting, finance, management science, productions and operations management22 Law, socio-legal studies and criminology23 Political science and public administration24 Philosophy25 Communication, cultural studies and women's studies26
golden rules of peeradjudicated grantmaking •The process is objective. It does not matter whether you have friends or colleagues on selection committees. •The process is not random. It is not a form of lottery. •Winning proposals are not selected. – weaker proposals are eliminated from the competition – the winners are those that remain.
The ‘rules of the game’ perpetuate the Matthew Effect “Unto he that hath shall be given. From he that hath not shall be taken away” i.e. the funding mechanism obeys a law of accumulated advantage
Scoring formula • Regular scholar – Record of achievement 60% – Research Program 40% • New scholar – Whichever is higher: • Record of achievement 60%, research program 40% • Record of achievement 40%, research program 60%
Research achievement • evaluation of the record of research achievement is based primarily on contributions to research the applicant has made within the last six years • if the applicant's research career has been interrupted, research achievement is evaluated based on his or her most recent period of research activity. • For regular scholars, applicant's five most significant contributions are taken into account in order to accurately situate the most recent six years in the context of the applicant's overall career. Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Research achievement Research contributions include: • refereed publications, including books, chapters of books and articles; • book reviews by the applicant/co-applicant or published reviews of his/her work; • research reports, papers presented at scholarly meetings or conferences, and other forms of written scholarly expression or participation in public discourse and debate which constitute a contribution to research; • where appropriate, contributions to the training of future researchers, including the supervision of graduate theses and/or the involvement of students in research activities; • research results from previous research grants, other awards from SSHRC or other sources; • academic awards and distinctions-new scholars may include scholarships and fellowships Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Research achievement Evaluation criteria: • quality and significance of published work (taking into consideration the quality of the chosen publication venues); • originality of previous research and its impact on the discipline or field; • quantity of research activity relative to the stage of the applicant's career; • demonstrated importance of other scholarly activities and contributions; • recentness of output (taking into account the nature of the applicant's career pattern and previous non-research responsibilities); Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Research achievement Evaluation criteria: • importance and relevance of dissemination of research results to non-academic audiences (as appropriate); • significance of any previous research supported by SSHRC or any other agency; • where applicable, contribution to the training of future researchers. (The committee must make allowances for applicants who have not supervised graduate students simply because their university does not offer graduate programs.) • efforts made, where appropriate, to develop research partnerships with civil society organizations and government departments. Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Record of research achievement
Research program: one or more projects over 3 years • explicit objectives, situated within the context of current scholarly literature; • relationship of the proposed research to the individual's ongoing research or to insights gained from earlier achievements-, • importance, originality and anticipated contribution of the proposed research; • theoretical approach or framework; • research strategies or methodologies (detailed methodology not necessary); Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Research program • plans for the communication of research results within and beyond the academic community • specific roles and responsibilities of students and research assistants, including how their duties will complement their academic training; • relationship of requested budget to proposed program of research. Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Research program: evaluation criteria • degree of originality and nature of expected contribution to the advancement of knowledge • scholarly and intellectual as well as social and cultural significance of the research • appropriateness of the theoretical approach or framework; • appropriateness and expected effectiveness of the research strategies or methodologies • suitability and expected effectiveness of plans to communicate research results both within and, as appropriate, beyond the academic community Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Program of research
Score needed for funding Meritorious but not funded cutoff zone: currently about 7.3 for SRGs funded rejected Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Achievement vs. research program total score 9 8.5 8 7.5 achievement 6 funded achievement 7 7 6.5 achievement 8 Meritorious but not funded 6 5.5 5 achievement 9 rejected 5 6 7 8 9 research program Source: SSHRC SRG Program Manual for Adjudication Committee Members, Dec. 2001
Some common errors • Theoretical framework weak or insufficiently explained • Methodology weak or insufficiently explained • A project is extended over 3 years to make it look like a program • Budget is padded or poorly formulated • Padding of CV • “me too” proposals – SSHRC funded research like this last year – Another research project in already worked-over area • Implausible teams
Some common errors • Failure to respect page limits (6 pages means 5.75-6 pages – 6.1 pages is no good) • Include literature review or information compilation as research • Grad students’ roles not consistent with research program flow and objectives • In a resubmission, failure to take into account the views of the committee and the external assessors • Ultra cartesian or ultra baconian research designs
Risky storylines • “I’m Too Important to Submit a Fully Worked-out Research Proposal – my record speaks for itself” – Variant: “We’re a team of Very High Profile Researchers. Our collective Research Achievement is off your scale” • “The fate of the world hangs on the outcome of my project”
Risky storylines • “My colleague got a grant last year to work on hamsters, so I will work on hamsters also” • “It would please God if this proposal were funded” • “My research results will overturn all established theories” • “The Minister mentioned that this would make a great research project” • “Because of the proliferation of incommensurable discourses in late postmodernity, you cannot understand what I am saying and I cannot understand my respondents, but I will study them anyway if you pay for it”
A typical 3-year program • Year 0: literature review completed; methods and instruments selected; preliminary hypotheses formulated • Year 1: refine instruments and hypotheses through qualitative research (focus groups, grounded theory, etc.). Test instruments • Year 2: apply instruments for data gathering • Year 3: analysis, interpretation, modeling, dissemination of results
Common winning formulas for new scholars • New scholar with good track record extends doctoral research via 3-year program – Watch out. If the proposed research is too close to the doctoral research, it will be regarded as derivative. If it is too far away, it will be regarded as too bold. – The most successful ones have published several articles (often with their PhD supervisor) before applying for a first grant
Common winning formulas for new scholars • New scholar as PI with established scholar as co-investigator with specified roles – The co-investigator brings up the “research achievement” score in proportion to his/her role in the project
Budgeting tricks and traps • the committee may reduce your requested budget. • It is good to ask for money for grad student stipends – build grad students into your program – Note: It is best to use doctoral students. In regional universities it is OK to use master’s students. If you use undergrads, make sure you have a good reason.
Budgeting tricks and traps • Do NOT request conference travel money in Year 1. – Hint: OK to request modest funds for grad student travel to conferences, if they present. • Do NOT inflate travel cost estimates. – it is permissible to include travel costs of work with research collaborators, but not collaborators’ research costs • Note that research travel costs include dissemination costs, which are also calculated separately
Budgeting tricks and traps • Do NOT request funds for computers unless computers are clearly necessary for the research and they are unavailable through the university – OK to ask for laptops for field research • NEVER ask for funding for less than three years
Budgeting tricks and traps • Research Time Stipends are only available if the home university contributes one-to-one matched funding • Do NOT request funds for books. SSHRC does not like to pay for books. • Be CAREFUL if you request funds for consultancies – this is thin ice • NOTE THAT once the budget is approved, SSHRC says that you can spend your grant however you like – but your University controllers do not necessarily know this. – At any rate, you cannot pay yourself an honorarium.
Budgeting tricks and traps • See SSHRC’s list of ineligible items. It includes: training, purchase or rental of standard office equipment, preparation of teaching materials, entertainment and hospitality costs, research leading to a degree, fees and honoraria to colleagues, indirect or overhead costs, etc. • ALWAYS include a clear explanatory budgetary note
Budget for hypothetical three-year, one-person small project at a small or medium (non-doctoral) university year 1 Master's year 2 year 3 12000 12000 12000 3800 3800 3800 travel-applicant Canadian foreign 1000 2000 2000 3000 2000 travel-student Canadian foreign 2000 RTS other expenses professional supplies 2000 equipment 4000 other equipment 1000 other 1000 total 26800 1500 10,000 1000 1000 1000 1000 31800 24300
Thank you! Questions or comments?
SSHRC Fellowship Application ... What happens when you submit your application 3. How are you evaluated 4. ... proposals from about 10 different ...
... and to consider addressing one or more of these areas in their research proposal. ... SSHRC monitors research ... SSHRC Doctoral Fellowships ...
SSHRC Research Proposal Guidelines . Detailed Description . Write your proposal in clear, ... Describe the proposed research strategies/key activities, ...
SSHRC grant applicants submit detailed research proposals that are evaluated: ... The review committees recommend to SSHRC which proposals to fund, ...
The Social Sciences and Humanities Research Council (SSHRC) ... their intended research proposal to: doctoral ... SSHRC Doctoral Award applications are ...
Argumentative “moves” in SSHRC proposals (see last slide for sources on this) ... research teams . Informal Argument and SSHRC = CRSH Academic Writing
... or research proposal ... and Humanities Research Council of Canada (SSHRC) ... being evaluated on your ability to research and write an ...