AIMS

Open Scholarship Output, Evaluation and Effects ... towards Responsible Metrics

(Image source: Knowledge Exchange approach towards Open Scholarship, KE, 2017 (*))

Within the shift toward Open Scholarship - "here [*] intended as an inclusive synonym for Open Science, Open Research, Science 2.0 and other similar terms" - we need deeper understanding of HOW the MOTIVATIONS of different actors are affected by the interactions between system (MACRO), individual (MICRO) and organisations (MESO) ... 

Practical interventions intended to IMPROVE the PROCESS OF EVALUATION are rarely assessed for their EFFECTS ... To move beyond there is a "need to fix the INCENTIVES” to questions of HOW and WHERE this can be done, and this will require a much greater understanding of how evaluation systems affect both the behaviour of individuals and the culture of communities. 

Knowledge Exchange (KE) aims to take on activities that make Open Access and Open Scholarship more easily understandable and approachable, and thus to contribute to the establishment and enhancement of a well-functioning Open Scholarship eco-system. 

To inspire and advise KE on the work in the Open Scholarship area, the recently formed  KE Open Scholarship Advisory Group (KEOSAG) will act as a "think tank plus" group of experts (from within and outside the partner organizations) to examine - from an integrated perspective - progress towards Open Scholarship and challenges of its implementation. 
KEOSAG's expertise and ideas will help initiate and shape activities that KE will undertake. Outcomes of these activities will be analysed and evaluated. 

While being particularly focused on two areas that are of special interest to the KE partners: 

(1)  the Economy of Open Science, and 
(2)  Output and Evaluation from the Researcher's Perspective,

a recently released (August 2017) KE's Report entitled : "Knowledge Exchange approach towards Open Scholarship":

  • Describes the present situation of Open Scholarship,
  • Addresses a range of challenges and opportunities for progress towards Open Scholarship, 
  • Presents the motivations for this latter and
  • Proposes the navigation towards the realisation of the Open Scholarship eco-system. This is possible through the outlined idea of a KE Open Scholarship Framework (with recommendations proposed), which helps to Identify the Arenas (political -- economic -- social -- technology) and the phases of the Research Life Cycle where work needs to be done to better navigate towards Open Scholarship.

Highlights from Chapter 5: Output and Evaluation from the researcher’s perspective

A key question is how culture affects (and effects) behaviour, and how external regulation and assessment affects culture. As this is a large topic, it is proposed to focus on two elements of the research life cycle in the first instance:

(1.) Outputs, such as articles, books, preprints, protocols, software, shared data, etc. and the
(2.)  Evaluation of these outputs, their outcomes and impacts. 

Having a thorough understanding of HOW RESEARCHERS FEEL about the current systems of RESEARCH EVALUATION, including their attitude towards the impact of Open Scholarship in this field, would help in:

  • identifying priority areas for improvement and support, and
  • improving the much needed close cooperation between (among others) researchers, librarians, research managers, leaders of academic institutions and policy makers. 

"We cannot address research evaluation without engaging  in the debate in this field over “METRICS” and INDICATORS, particularly QUANTITATIVE INDICATORS..."

"What kind of support do researchers need in evaluating their own societal impact and how does evaluation affect their practice?"

"Does Open Scholarship actually need INDICATORS? For whom, and for what reason? What does it mean and what should it actually mean to researchers? One may also question current funding practice: what research needs to be funded and how should research evaluation be linked to research funding?"

"... INCENTIVES for QUANTITATIVE EVALUATION require re-thinking as well... should they drive new forms of evaluation and indicators? What is currently being evaluated and why? The rising importance of data, and in the future of research data, may be an opportunity to explore this".

"How may current INDICATORS become RESPONSIBLE and Open Scholarship - aware?"

"How may “RESPONSIBLE METRICS*” capture the progress towards full Open Access (by 2020) to research publications? Or can they?"

[see: Recommendations on Open Science Publishing Adopted by the Open Science Policy Platform on 25 April 2017]

"How may RESPONSIBLE EVALUATION capture Open Scholarship contributions that are currently ignored? ..."

"How may RESPONSIBLE and Open Scholarship-aware EVALUATION SYSTEMS be used to generate INCENTIVES and REWARDS?"
 

Take a look at ‘Knowledge Exchange approach to Open Scholarship' and find out how these and other more complex and interdisciplinary questions and overwhelming growth in data generation provide opportunities and challenges for better research and its evalutation ….

Overall, the report can be read like a roadmap for the transition towards Open Scholarship and seen as the enhancement of the past KE Conference "Pathways to Open Scholarship".

Source: KE Approach to Open Scholarship.

* “RESPONSIBLE METRICS can be defined by the following key principles:

  • Robustness – basing metrics on the best possible data in terms of accuracy and scope
  • Humility – recognising that quantitative evaluation should support, but not supplant, qualitative, expert assessment
  • Transparency –that those being evaluated can test and verify the results
  • Diversity – accounting for variation by research field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system
  • Reflexivity – recognising and anticipating the systemic and potential effects of indicators, and updating them in response”, - Forum on Responsible Metrics

 

Topics: