We live in the society of evaluation that permeates all aspects of academic and intellectual activity. Quantitative assessments of the activities carried out by scholars proliferate going hand in hand with initiatives and manifestos that highlight their limitations and shortcomings. In this project, we address the issue of "right evaluation" keeping a neutral approach, i.e. without having ideological prejudices for or against the evaluation itself.
This ambitious objective will require overcoming a number of theoretical and methodological challenges. They include:
i) Development of an ontological framework to include the concepts related to "right evaluation" and "right metrics";
ii) Development of a comprehensive and multidimensional performance model as a base for robust empirical analysis;
iii) Integration of existing datasets with interviews and a questionnaire to account for scholars' psychology and motivation;
iv) Development of new methodologies to estimate the performance of scholars within their institutional context.
Despite the relevance of this topic for research and educational policy-making, there are very few contributions on this issue.
To face these challenges we will adopt an interdisciplinary and qualitative-quantitative multi-methods approach combining virtue ethics and accounting with efficiency analysis and machine learning.
We will try to demonstrate that a "right evaluation" cannot be separated from a good evaluation and that to make a good evaluation it is necessary to go beyond the existing quantitative measures currently used but also beyond the superficial and ideological criticisms towards evaluation. When evaluating one must evaluate well, that means considering all costs, including moral ones, of the evaluation and who has to pay for them.
WHAT THIS PROJECT WILL ACCOMPLISH THAT IS NOT BEING DONE IN THE EXISTING LITERATURE
The first accomplishment of this project relates to the development of the notions of "right evaluation" and "right metrics" to complement those of "responsible evaluation" and "responsible metrics" analysed in the existing literature recalled above.
The inclusion of these concepts in our evaluative framework will allow us to operationalize the notion of > and > evaluation of research practices making a step further in the characterization and assessment of the performance of individuals involved in the research practices and institutions who include them.
The development of this ontological framework will provide a multi-stakeholder model of performance, that will enable us to assess the sensitivity and robustness of metrics in a multidimensional framework.
The investigation about how rankings can benefit from the inclusion of financial data will provide the European policy makers with helpful suggestions to promote a path to accounting harmonization for the university public financial reports.
The empirical analysis based on interviews and questionnaire for the assessment of the virtues of researchers will provide a Sapienza's flagship study on "Right Metrics" that will highlight Sapienza's values and human capital offering a model to characterize university research practices that may be extended to Italian, European and worldwide universities.
The methodological research based on the combination of efficiency analysis and machine learning will provide an updated technical approach to generalize currently used efficiency models, moving from efficiency (based on the outputs to inputs relationships) to the effectiveness (including the objectives of the units in the analysis) accounting for impact and providing a sustainable evaluation approach.
HOW WILL THE PROJECT CHANGE OUR UNDERSTANDING ABOUT EVALUATION AND METRICS
This project will provide a useful framework for the "good" and "right" evaluation of research practices.
Our research will offer a self-assessment tool for researchers, to understand the functions of their research activities, their motivations and where they are in their research practice.
It will help institutions to collect and describe the main functions of the research practices (highlighting their special features) developed by its researchers, and their motivations, to include them in their strategic plan.
To conclude, this project has the potential of introducing a "paradigm shift" in the evaluation of research practices. From an evaluation focused on the
Finally, the results achieved by this project may be the starting point for the preparation of an Advanced ERC grant proposal on the issues investigated within this project.