Published on February 16, 2014
The impact of evaluations on developments in academic careers Frank van der Most DANS and the e-Humanities Group ! ACUMEN meeting, 9 - 11 January 2013 1
ACUMEN Academic Careers Understood through Measurement and Norms www.research-acumen.eu ! “Currently, there is a discrepancy between the criteria used in performance assessment and the broader social and economic function of scientiﬁc and scholarly research.” (proposal, p. 3) !2
ACUMEN ✦ “... broader social functions of the scientiﬁc system are often not included in its quality control mechanisms.” (proposal, p. 3) ✦ so, ACUMEN will develop •a portfolio for researchers, and •criteria and guidelines for Good Evaluation Practices !3
Research Question ✦ How do evaluations inﬂuence the development of individuals’ careers in academia? (What is the impact of evaluations?) •Use perspective for ACUMEN ✦ Concepts •Wide notion of evaluations •Careers + ‘work’ !4
Project partner Country People Universiteit Leiden Netherlands Paul Wouters, Clifford Tatum, Laura de Ruiter Bar Ilan University Israel Judit Bar-Ilan, Hadas Shema Consejo Superior de Investigaciones Cientificas (CSIC) Spain Isidro F. Aguillo, Mario Fernández Pérez, Amalia Mas Bleda University of Wolverhampton United Kingdom Mike Thelwall, Kayvan Kousha Sihtasutus Archimedes Estonia Ülle Must, Arto Mustajoki, Marcus Denton, Taavi Tiirik, Oskar Otsus Humboldt-Universität zu Berlin Germany Stefan Gradmann, Frank Havemann Technische Hochschule Wildau Germany Rainer Voss, Hildrun Kretschmer, Ulla Große, Martina Brandt, Frank Hartman, Dana Mietzner Danmarks Biblioteksskole Denmark Birger Larsen, Jesper W. Schneider, Lorna Wildgaard Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) Netherlands Andrea Scharnhorst, Sally Wyatt, Frank van der Most
Who does what? Work Package Who 1 Evaluation impact KNAW, Archimedes 2 Institutional Web Presence University of Wolverhampton, CSIC 3 Researchers’ Web image Bar Ilan University 4 Gender effects of evaluation Technische Hochschule Wildau 5 New bibliometric indicators Humboldt-Universität ,Danmarks Biblioteksskole 6 ACUMEN portfolio All 7 Management Universiteit Leiden !6
Impact A change in a career Evaluatio F.V. van der Most, 2012 !7
Impact Event A Event U A change in a career Evaluatio nt D F.V. van der Most, 2012 Event B Event V Event W Micro story Event C !7
Method ✦ Quite structured, in depth interviews ✦ 4 Countries : UK, Germany, the Netherlands, Poland ✦ 4 Disciplines : Astronomy/astrophysics, Environmental engineering, Philosophy (+), Public Health ✦ 3 levels of seniority: PhD + 5 to 10 years, + 11 to 25 years, + 26 years or more (or equivalents) !8
Method: interview protocol ✦ What are the three most important developments in your career? ✦ What are the three most inﬂuential evaluations in your career? !9
Work in progress Number of interviews done AA EE P+ PH Total UK 4 (3+1) 4 (3+1) 3 (2+1) 4 (3+1) 15 (11+4) Germany 2 (2+0) 3 (3+0) 3 (2+1) 3 (3+0) 11 (10+1) The Netherlands (ongoing) 1 (1+0) 2 (1+1) 1 (1+0) 3 (2+1) 7 (5+2) Poland (feb 2013) - - - - Numbers in brackets: individuals + deans/HR managers !10
Indicator, what is the overlap? Don’t know exactly, yet. !11
Basic answers: 3 most inﬂuential evaluations Coding group Total Job applications 21 Grants 14 Job-related evaluations (not appl.s) 11 Diplomas 6 Other 5 Paper reviews 5 Informal evaluations (other) 3 Evaluations leading to invitations 3 Feedback 3 Evaluation of teaching 1 Total 72 !12
Basic answers: 3 most inﬂuential evaluations Coding group Total !13
Basic answers: 3 most inﬂuential evaluations Coding group Total Job applications 16 !13
Basic answers: 3 most inﬂuential evaluations Coding group Total Job applications 16 Grants 14 !13
Basic answers: 3 most inﬂuential evaluations Coding group Total Job applications 16 Grants 14 Informal evaluations 12 !13
Basic answers: 3 most inﬂuential evaluations Coding group Total Job applications 16 Grants 14 Informal evaluations 12 Diplomas 6 Other 5 Paper review 4 Job-related evaluations (not appl.s) 3 Evaluations leading to invitations 1 Evaluation of teaching 1 Total 62 !13
Examples: informal job applications !14
Examples: informal job applications ✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH) !14
Examples: informal job applications ✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH) ✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE) !14
Examples: informal job applications ✦ A MA student’s supervisor offers a job on a project leading to a PhD (DE AA, DE PH) ✦ An informal conversation leads from a temporary to a permanent position as lab manager (UK EE) ✦ An earlier conversation later on appears to be an important element in a job application (UK AA) !14
Examples: evaluations leading to invitations !15
Examples: evaluations leading to invitations ✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH) !15
Examples: evaluations leading to invitations ✦ A HoD asks the interviewee to lead a project, a great opportunity for a relatively junior researcher (UK PH) ✦ An editor asks the interviewee to become the successor (UK PH) !15
Examples: other ✦ Course applications (BA to PhD) (2) ✦ ‘Still to be determined’ (1) ✦ Winning a prize (1) ✦ Examinations during a study (1) !16
Relevance for governance of science ✦ Evaluations seem to inﬂuence careers (but further analysis needed) ✦ Share of informal evaluations may be more inﬂuential than expected •The ‘system’ is not so ‘systematic’ or not so transparent •Sometimes systematically opaque (Academic freedom) !17
Thank you for your attention! ! ! Questions, suggestions? ! ! ! www.researchaffairs.nl
John's presentation to EMBA students on knowledge work, the innovation ethics and ...
Looking for an icebreaker that helps team members learn more about each other, imp...
Quantas vezes você pensou sobre diferentes formas de conseguir ter mais foco. São ...
Os segredos do sucesso - Dale Carnegie O livro de Dale Carnegie "Como Ganhar Ami...
The impact of evaluations on academic careers. An study of academics in four disciplines, four countries and three cohorts. Research output: ...
1.2 Academic careers and the ... Citation patterns are strongly linked to academic career development, ... academic impact for those disciplines ...
Educational Impact Evaluation. SRI's education researchers conduct objective impact evaluations ... Teacher Professional Development Can Make a Positive ...
• Formative evaluation • Personal development ... • Academic knowledge • College/ career ... impact academic outcomes,
... program performance and impact. USAID uses evaluation ... public evaluations, visit the Development ... career opportunities at ...
... Lessons and Implications of the Latest Research on Credit: Academic ... these evaluations shed new light on the impact of ... and Development, ...
The Impacts of Corporal Punishment on Students’ Academic ... persevering to vacate the roots of personality development and academic career of the ...
IIE Center for Academic Mobility Research and Impact. ... Program in supporting women to pursue academic careers in ... and impact evaluations for ...