Driven by UHN’s mission for research excellence, Dr. Anna Gagliardi, Senior Scientist at Toronto General Hospital Research Institute (TGHRI), led an advisory group to redefine research assessment practices at UHN.
In January 2020, UHN signed the San Francisco Declaration on Research Assessment (DORA). The agreement outlines principles that promote better approaches to assessing research impact than the traditional overreliance on journal and publication metrics.
“This commitment marked an important milestone,” says Dr. Brad Wouters, UHN Executive Vice President of Science and Research. “We are changing research assessment in ways that provide a more complete understanding of our researchers’ work in order to drive continuous improvement in research quality and impact.
“This is done as part of our larger strategy to advance our vision of A Healthier World.”
The DORA Advisory Group, which included researchers from across UHN, was created to put the principles outlined in DORA into action.
Two recent publications outline their approach and progress – one published in PLoS One and the other in Health Research Policy and Systems – and are the first publications featuring researchers from across all six UHN research institutes.
“Initially, we focused on identifying and prioritizing measures for assessing research performance according to the DORA principles,” says Dr. Gagliardi. “These measures would help with hiring, annual review, promotion and tenure decisions.”
Through a comprehensive review of literature, the group compiled an initial list of 50 ways of assessing research activity and grouped them into eight domains. These domains included but were not limited to research productivity, strategic value, funding, publications, innovations and impact.
“To consolidate and prioritize these measures, we conducted two rounds of surveys with a panel of diverse researchers,” says Himani Boury, a previous practicum student at the Toronto General Hospital Research Institute and first author on the Health Research Policy and Systems publication.
The findings led to the identification of the following ten priority research assessment measures:
High Importance
Advancing existing knowledge
Innovative research plans
Independent funding support
Relevant research outputs
Publication content quality
Moderate Importance
Challenges to research productivity
Potential to improve health or health care
Team science
Collaboration
Recognition by professional/academic groups
“Our next focus was exploring the potential barriers to implementing and adopting these measures on an organizational level,” says Himani. “To do so, we conducted 18 interviews with researchers and research support staff from across UHN’s six research institutes.
“This shed light on a several strengths and barriers to implementation.”
Participants found the measures to be comprehensive, relevant across disciplines and rigorously developed. The reporting template for the measures was also deemed easy to understand and use. However, concerns were expressed over the measures’ relevancy across research disciplines. Another concern raised was that the complexity of the measures would make them more time-consuming to use.
To overcome these barriers, various strategies were suggested, including high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting tools, guidance for evaluators, and sharing of approaches among research institutes.
“Addressing barriers and implementing supportive strategies is crucial for the adoption of prioritized measures. These findings urge research organizations worldwide to reassess evaluation practices, embrace DORA-compliant measures, and work collaboratively to foster a transformative shift towards prioritizing quality and impact in research assessment,” concludes Dr. Gagliardi.
The work of the DORA Advisory Group will be continued by the Research Assessment Implementation Committee led by Drs. Michael Hoffman and Gagliardi.
Read more about this research.
This work was supported by UHN Foundation and The Princess Margaret Cancer Foundation.
