Phase I (2003-2006)
Within several sub-projects, the DFG Research Group "Internationale Wettbewerbsfähigkeit und Innovationsfähigkeit von Universitäten und Forschungsorganisationen - Neue Governanceformen" (International Competitiveness and Innovativeness of Universities and Research Organisations - New Governance Forms) (Ref: FOR 517), analysed the connections between governance and performance. The fundamental question posed was which indicators are suitable for measuring performance and how (best) to collect them. In the present project, patent and publication indicators were generated on the one hand and placed at the disposal of the other members of the Research Group. In addition, suggestions for further indicators were worked out, in order to find out which indicators or combinations thereof are particularly meaningful and applicable in practice. The question was also raised to what extent different disciplines require different indicators.

The analysis was mainly focused on the characteristics of co-publication and third-party funding indicators to measure scientific performance.

Phase II (2006-2009)
In the second project phase a dataset was collected on the work, production and governance conditions of German research units (university chairs and groups in non-university research working along thematic lines) from astrophysics, nanotechnology, biotechnology and economics (n=473). Based on this dataset, answers to the following questions in particular were sought:

• How do the university reforms influenced by the New Public Management approach affect research performance?
• What do "production structures" in science look like?
• Do system-immanent disincentives exist in the production structures and can they be corrected by means of indicator-supported grants/ allocations?
• What do "good" indicator systems of performance measurement in the sciences look like? Which dimensions of scientific capability must be taken into account?

Latest Results
Main results of Phase I referred to the indicators international co-publications and third-party funds. It transpired that, in contrast to frequently expressed assumptions, co-publications (and thus internationality) are no indication for the quality of research. They therefore appear questionable as indicators of performance measurement and rather reflect an altered behavioural communication pattern in the sciences (Schmoch/Schubert 2008). Third-party funding has a performance-enhancing effect, but only up to a subject-specific limit. Above this limit, the third-party quotas reduce performance (Jansen et al. 2007; Schubert 2008). In Phase I it was also demonstrated that scientific performance is multi-dimensional. In particular, a specialisation of the research units in the three areas generation of new knowledge, education of junior scientists plus knowledge transfer and infrastructure services for science can be observed (Jansen et al. 2007; Schubert 2007).

The main focus in Phase II was primarily concentrated on judging the impact of the university reforms. It emerged that through the growing orientation towards the New Public Management approach, potentials can indeed be exploited which promise a more efficient utilisation of resources in science (Schubert 2007, 2008, 2009). This however has less to do with the much discussed "laziness" of the professors than with system-immanent (suboptimal) incentive-schemes, which have their roots in the interdependencies in the scientific production process (Schmoch et al. 2007; Schubert/Schmoch 2008). In particular, sub-optimal provision of funds to promote up-and-coming scientists and infrastructure services are apparent, such as activities in academic self-administration, editing expert scientific journals as well as knowledge transfer to politics and business. The consequence for indicator-supported fund allocation as one of the most vital instruments is clear: these disincentives can be corrected by means of financial incentives. The recommendation is therefore to collect a limited number of indicators (not more than 10-12) on a comparable basis from the universities as a standard, in order to arrive at national benchmarks on a stable and statistically reliable basis. For this purpose, the three dimensions generation of new knowledge as an original research activity, education of junior scientists, as well as knowledge transfer and infrastructure should be taken into account. Contrary to current practice, third-party funds should be awarded only up to a certain discipline-specific limit. A further consequence from the multi-dimensionality of scientific performance can be seen in the fact that even the top-class researchers also have to rely on the preliminary inputs of their colleagues (e.g. in the form of well educated scientific staff). One must not overlook the academic midfield players. Or in other words: even "lighthouses" need a solid foundation to stand on. Indicator systems can be applied as a relatively cost-effective screening instrument in order to identify, on the one hand, excellent and, on the other hand, seriously sub-average performance in research. Taking into account the fact that the midfield players' performance is the foundation on which top-class research is built, redistribution on a larger scale should take place only between the really good and the really bad units. The indicator-based screening process should be supplemented, if necessary, by a peer-review procedure to validate far-reaching budget decisions. This would also have the advantage that the "experts", always in short supply, can be requested to write expertises only in really critical cases (Schmoch 2008).
 

Status:
ongoing (10.2006-09.2009)

Client:
German Research Association (DFG)

Publications:

Schubert, T. (forthcoming): Empirical Observations on New Public Management to Increase Efficiency in Public Research – Boon or Bane? Research Policy.

Schmoch, U. (2009): Geeignete Ansätze zur Messung wissenschaftlicher Leistung, in: Beiträge zur Hochschulforschung, Vol. 31 (1) 31, pp. 26-41.

Schubert, T.; Sooryamoorthy, R. (forthcoming): Can the Centre-Periphery Model Explain Patterns of International Scientific Collaboration among Threshold and Industrialised Countries? The Case of South Africa and Germany, Scientometrics.

Schmoch, U.; Schubert, T.; Jansen, D.; Heidler, R.; van Görtz, R. (forthcoming): How to Use Indicators to Measure Scientific Performance? A Balanced Approach. Research Evaluation.

Schubert, T.; Grupp, H. (2009): Tests and Confidence Intervals for a Class of Scientometric, Technological and Economic Specialisation Ratios. Applied Economics, forthcoming.

Schmoch, U.; Schubert, T. (2009, im Erscheinen): Sustainability of Incentives for Excellent Research - The German Case. Scientometrics.

Schubert, T. (2008): Wissenschaftsfreiheitsgesetz: Richtiger Weg. Wirtschaftsdienst - Zeitschrift für Wirtschaftspolitik, Vol. 88 (8), p. 490.

Schmoch (2008): Fachspezifische Indikatoren formulieren: Geeignete Ansätze zur Messung wissenschaftlicher Performanz, in: Kehm, B.; Mayer, E.; Teichler, U. (Eds.) Hochschulen in neuer Verantwortung. Strategisch, überlastet, divers? Bonn: Lemmens Medien, pp. 272-274.

Schmoch, U., Schubert, T. (2008): Nachhaltigkeit von Anreizen für exzellente For-schung, in: Hornbostel, S.; Simon, D. (Eds.), Exzellente Wissenschaft. Das Problem, der Diskurs und die Folgen. iFQ-Working Paper No. 4, Bonn, pp. 39-49.

Schmoch, U.; Schubert, T. (2008): Wie gut ist der Standort Südwest? Bild der Wissenschaft plus, Der große Wurf: Nanotechnologie. Sonderbeilage zu Bild der Wissenschaft, 6/2008, pp. 28-29.

Schmoch, U./Schubert, T. (2008): Are International Co-publications an Indicator for Quality of Scientific Research? Scientometrics, Vol. 74 (3), pp. 361-377.

Schubert, T./Schmoch, U. (2008): How Lazy Are University Professors Really: A Not So Seriously Meant Note on Observations Made During an Online-inquiry. Soziale Welt, Vol. 59 (1), pp. 75-78.

Jansen, D./Wald, A./Franke, K./Schmoch, U./Schubert, T. (2007): Third Party Research Funding and Performance in Research. On the Effects of Institutional Conditions on Research Performance of Teams, Kölner Zeitschrift für Soziologie und Sozialpsychologie, Vol. 59, pp. 125-149.

Schmoch, U./Schubert, T./Jansen, D. (2007): Suitability of Indicators for Measuring Scientific Performance, unpublished manuscript.