Hans Dieter Daniel

38 %
63 %
Information about Hans Dieter Daniel
Education

Published on February 28, 2008

Author: Perrin

Source: authorstream.com

Slide1:  The potential and problems of peer evaluation in higher education and research Hans-Dieter Daniel Professorship for Social Psychology and Research on Higher Education, ETH Zurich Evaluation Office, University of Zurich Slide2:  “Peer review (known as refereeing in some academic fields) is a scholarly process used in the publication of manuscripts and in the awarding of funding for research. Publishers and funding agencies use peer review to select and to screen submissions. The process also forces authors to meet the standards of their discipline and thus achieve scientific objectivity. Publications and awards that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals in many fields”. Source: Wikipedia, The Free Encyclopedia Critics of peer review:  Critics of peer review "Manuscript refereeing, one aspect of peer review and self-management in the sciences, has been shown to be almost wholly lacking in interreferee agreement on the recommendation to publish (r2 = .04), without validity in forecasting the subsequent usefulness of a work to scientists as reflected in citations of the work in other scientific papers (r2 = .00), and biased in more than a dozen ways." (Ross, 1980) "Peer review fails miserably with respect to every technical criterion for establishing the reliability and validity of an assessment instrument." (Bornstein, 1991) Reliability: Agreement of referees in journal peer review:  Reliability: Agreement of referees in journal peer review Fairness: Gender bias in grant peer review:  Fairness: Gender bias in grant peer review Bias in favour of females Bias in favour of males Predictive validity: Comparison of the citation rates for papers accepted and rejected by The Journal of Clinical Investigation but published elsewhere (source: Wilson, 1978):  Predictive validity: Comparison of the citation rates for papers accepted and rejected by The Journal of Clinical Investigation but published elsewhere (source: Wilson, 1978) Mean number of citations of articles previously published to approval or rejection by B.I.F. post-doctoral applicants:  Mean number of citations of articles previously published to approval or rejection by B.I.F. post-doctoral applicants Notes. Application window: 1990-1995; publication window: 1986-1994; citation window: from year of publication to the end of 2001. We included only average numbers of citations per year that could be calculated with citation counts for more than ten articles (1986–1994); the years 1981 to 1985 and 1995 to 1996 could not be included. Slide8:  Folie Evaluationsverfahren Academic Departmental Review at the University of Zurich University of Zurich: a member of LERU:  University of Zurich: a member of LERU Members of the League are Universiteit van Amsterdam Lunds universitet University of Cambridge University of Milan University of Edinburgh LMU Munich Universität Freiburg University of Oxford University of Geneva Université Pierre et Marie Curie, Paris 6 University of Heidelberg Université Paris-Sud 11 University of Helsinki Karolinska Institutet, Stockholm Leiden University Université Louis Pasteur Strasbourg Katholieke Universiteit Leuven Universiteit Utrecht University College London Universität Zürich Membership in the League of European Research Universities (LERU), which is by invitation, is periodically evaluated against a broad set of quantitative and qualitative criteria, such as research volume, impact and funding, strengths in PhD training, size and disciplinary breadth, and peer-recognised academic excellence. Citation analysis in science and medicine of 40 major European universities, 1995-1999:  Citation analysis in science and medicine of 40 major European universities, 1995-1999 Source: Centre for Science and Technology Studies (CWTS), Universiteit Leiden CPP: average number of Citations Per Publication, JCS: mean Journal Citation Score of the institute‘s ‘journal set‘, FCS: mean citation rate of all papers published in all journals of the field(s) in which the institute is active Organizational Structure (extract): Independent Evaluation Office:  Organizational Structure (extract): Independent Evaluation Office Evaluation Office Board of the University Minister of Education, and six personalities from science, culture, business, and politics Extended Executive Board Executive Board President, Vice-Presidents, Managing Administration Director Senate Supervision / Strategic Management Executive Management Evaluation Objectives:  Evaluation Objectives Assess, assure, and improve the quality of academic work in research, teaching, and services as well as assure the quality of management and administration (referred to as “Full Review” at the University of Cambridge) Provide decision aids to support medium and long-term strategic planning Report to the public (accountability) Evaluation Office:  Evaluation Office Personnel Director Prof. Dr. Hans-Dieter Daniel, ETH Zurich Managing Director Dr. Thomas Rothenfluh Project Managers Andrea Eichholzer, Judith Maag, Dr. des. Sandra Mittag, Vinciane Vouets, Peter Koch Secretary Annette Häusermann Projects ~113 evaluation projects within six years Costs 0,14 % of the University’s budget (including third party funds) (2004: env. 932 Million Swiss Francs) First evaluation cycle: Project status:  First evaluation cycle: Project status Evaluation Process: Informed peer review:  Surveys Students Alumni Academic Staff Personnel Customers Self-evaluation report Responses Unit under evaluation Evaluation Office Comprehen-sive Evaluation Report Responses Course participants Semester 1 Semester 2 Semester 4 Bibliometry Evaluation Process: Informed peer review Experts (Peers) Experts’ Report Site Visit Semester 3 Follow-up Board, Executive Board Memorandum of understanding / Goal agreement Self-Evaluation: Section Headings for Self-Evaluation and Guidance on Content:  Self-Evaluation: Section Headings for Self-Evaluation and Guidance on Content Retrospective on last five years History of the institute Structure and organization Financial resources: Budget and third-party funds Human resources Material resources (e.g., IT, library, equipment, spatial situation) Management and administration Teaching and study programs Research Promotion of young academics/scientists Services Internal quality assurance measures Profile of strengths and weaknesses Future perspectives Brief portraits of the chairs, senior lecturers, research group leaders Research (qualitative / quantitative); teaching; promotion of young academics / scientists; offices held / professional functions / services Selection criteria for external reviewers :  Selection criteria for external reviewers Each expert should have an outstanding international scientific reputation. The composition of the team of experts as a whole should ensure that the following requirements can be met: All core competencies of the unit under evaluation are covered Experience in the area of structured promotion of young academics / scientists (participation in graduate schools, PhD programs, etc.) Knowledge in the area of curriculum development Evaluation experience Experience in management / administration of sizeable university units (dean, vice president, assistant rector of a university) Active professors (Emeritae / Emeriti only in exceptional cases) Desirable: Representation of men and women; good age mix Familiarity with the Swiss university system External Reviewers: Bias:  External Reviewers: Bias Persons should not be selected as experts if there is any appearance of bias. Appearance of bias is given if a potential expert is in any way affected by participation in the evaluation or if there is any other reason that might render the impartiality of the evaluation suspect. The following can constitute such reasons: Teaching contracts or visiting professorships at the unit under evaluation Close scientific collaboration, e.g., joint projects or co-authorship of publications Teacher / student relationship Participation in ongoing or very recently concluded procedures for the appointment of professors in the environment surrounding the unit under evaluation Simultaneous or previous participation in boards or committees advising the unit under evaluation (e.g., Scientific Advisory Board) Personal or mutual economic interests; economic competition Informed (Evidence-based) Peer Review:  Informed (Evidence-based) Peer Review Slide20:  Lessons Learned from Departmental Reviews at the University of Zurich Strengths Evaluations find acceptance. They forge identity in the units under evaluation. They foster communication and transparency – within the university and with strategic and political authorities. They uncover strengths and weaknesses (and the mere announcement of a pending evaluation can in part contribute to performance improvement). They are indispensable for structure and development planning. Weaknesses Evaluation and implementing the results of the evaluation require a lot of work and are time-consuming for all participants. If human and equipment resources are found to be insufficient, the funds required can not always be secured from the university or the state. Meta-evaluation: Central Evaluation and Accreditation Agency Hannover (ZEvA) and Consortium of Universities in Northern Germany (Nordverbund) Overall assessment of the evaluation process by institute members and reviewers:  Meta-evaluation: Central Evaluation and Accreditation Agency Hannover (ZEvA) and Consortium of Universities in Northern Germany (Nordverbund) Overall assessment of the evaluation process by institute members and reviewers Meta-evaluation: Academic Advisory Council of Lower Saxony Quality assurance instruments for quality improvement of research and teaching (responses “very appropriate” and “appropriate”, in %):  Meta-evaluation: Academic Advisory Council of Lower Saxony Quality assurance instruments for quality improvement of research and teaching (responses “very appropriate” and “appropriate”, in %) Source: Academic Advisory Council of Lower Saxony (unpublished meta-evaluation report) Scenarios of future quality assurance in research (reviewers: n=54, faculty members: n=364; acceptance, in percent):  Scenarios of future quality assurance in research (reviewers: n=54, faculty members: n=364; acceptance, in percent) Source: Academic Advisory Council of Lower Saxony (Wissenschaftliche Kommission Niedersachsen) Publications:  Publications Bornmann, L. & Daniel, H.-D. (2006, accepted for publication). Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics. Bornmann, L., Mittag, S., & Daniel, H.-D. (2006, accepted for publication). Quality assurance in higher education - meta-evaluation of multi-stage evaluation procedures in Germany, Higher Education. Bornmann, L. & Daniel, H.-D. (2005). Does the h-index for ranking of scientists really work? Scientometrics, 65(3), 391-392. Bornmann, L. & Daniel, H.-D. (2005). Selection of research fellowship recipients by committee peer review. Analysis of reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics, 63(2), 297-320. Bornmann, L. & Daniel, H.-D. (2005). Criteria used by a peer committee for selection of research fellows. A Boolean probit analysis. International Journal of Selection and Assessment, 13(4), 296-303. Bornmann, L. & Daniel, H.-D. (2005). Committee peer review at an international research foundation. Predictive validity and fairness of selection decisions on post-graduate fellowship applications. Research Evaluation, 14(1), 15-20. Daniel, H.-D. (2005). Publications as a measure of scientific advancement and of scientists' productivity. Learned Publishing, 18(2), 143–148. Daniel, H.-D. (2004). Guardians of Science - Fairness and Reliability of Peer Review . Weinheim: Wiley-VCH. ISBN: 3-527-60220-8.

Add a comment

Related presentations

Related pages

Hans-dieter Daniel – CDs, Bücher, DVDs und mehr – jpc.de

Zahlreiche Attraktionen bieten die Küsten von Nord- und Ostsee und das sanft hügelige und seenreiche Binnenland naturbegeisterten Besuchern.
Read more

Hans-Dieter Daniel - Leitung Evaluation - Universität ...

XING ist Deutschlands größtes berufliches Netzwerk: Mit XING finden Sie Ihren Traumjob, knüpfen wertvolle Kontakte, tauschen Wissen aus – und haben ...
Read more

Hans-Dieter Daniel im Telefonbuch - Jetzt finden!

ll Hans-Dieter Daniel gesucht? Richtige Adressen und Telefonnummern finden! 7 Einträge zu Hans-Dieter Daniel mit aktuellen Kontaktdaten, Öffnungszeiten ...
Read more

Hans-Dieter Daniel im Handelsregister | Moneyhouse Deutschland

Im Handelsregister gibt es 9 Verbindungen zu Hans-Dieter Daniel: Bei welchen Firmen ist er Geschäftsführer? Welche gehören ihm?
Read more

Hans-Dieter Daniel SFT System- und Frästechnik

Hans-Dieter Daniel SFT System- und Frästechnik, Steyerberg | Firmenauskunft & Bonitätsauskunft | Branche: Erbringung von sonstigen Dienstleistungen a. n. g.
Read more

Hans-Dieter Daniel Videoproduktion in Biederitz Am ...

Hans-Dieter Daniel - Adresse: Am Weidenring 12 , 39175 Biederitz mit Telefonnummer +493929228861 und interaktivem Stadtplan. Rufen Sie Hans-Dieter Daniel ...
Read more

Hans Dieter Daniel Zank Installateurbetrieb, Duisburg ...

Hans Dieter Daniel Zank Installateurbetrieb. Risiken und Chancen erkennen mit der Creditreform Bonitätsauskunft; Details über Bonität, ...
Read more

Hans Dieter Daniel, Motorradhändler in Porta Westfalica ...

Hans Dieter Daniel, Lindenstr. 25 in Porta Westfalica Lerbeck, Telefon 0571/7798660 mit Anfahrtsplan
Read more

Hans-Dieter Daniel - Deutsche Digitale Bibliothek

Institutionelle Qualitätssicherung der Lehre auf dem Prüfstand : eine Fallstudie an der ETH Zürich Mittag, Sandra. - Bielefeld : UVW, Webler, 2012
Read more

Hans-Dieter Daniel - Außendienst Möbelbranche | XING

Die Position als Außendienstmitarbeiter in dem Gebiet Niedersachsen u. Bremen. Auch Bereichsübergreifend.
Read more