Redefining research impact at UHN

UHN Researchers and logo
Members of the DORA Advisory Group at UHN included, (clockwise from top left), Drs. Mathieu Albert, James CL. Chow, Ralph S. DaCosta, Anna Gagliardi, Michael Hoffman, Behrang Keshavarz, Pia Kontos, Jenny Liu, Mary Pat McAndrews and Stephanie Protze. (Image: UHN StRIDe Team)

Driven by UHN’s mission for research excellence, Dr. Anna Gagliardi, Senior Scientist at Toronto General Hospital Research Institute (TGHRI), led an advisory group to redefine research assessment practices at UHN.

In January 2020, UHN signed the San Francisco Declaration on Research Assessment (DORA). The agreement outlines principles that promote better approaches to assessing research impact than the traditional overreliance on journal and publication metrics.

“This commitment marked an important milestone,” says Dr. Brad Wouters, UHN Executive Vice President of Science and Research. “We are changing research assessment in ways that provide a more complete understanding of our researchers’ work in order to drive continuous improvement in research quality and impact.

“This is done as part of our larger strategy to advance our vision of A Healthier World.”

The DORA Advisory Group, which included researchers from across UHN, was created to put the principles outlined in DORA into action.

Two recent publications outline their approach and progress – one published in PLoS One and the other in Health Research Policy and Systems – and are the first publications featuring researchers from across all six UHN research institutes.

“Initially, we focused on identifying and prioritizing measures for assessing research performance according to the DORA principles,” says Dr. Gagliardi. “These measures would help with hiring, annual review, promotion and tenure decisions.”

Through a comprehensive review of literature, the group compiled an initial list of 50 ways of assessing research activity and grouped them into eight domains. These domains included but were not limited to research productivity, strategic value, funding, publications, innovations and impact.

Himani Boury, (L), a previous practicum student at the Toronto General Hospital Research Institute (TGHRI) and Dr. Anna Gagliardi, Senior Scientist at TGHRI
Himani Boury, (L), a previous practicum student at the Toronto General Hospital Research Institute (TGHRI), was first author on the Health Research Policy and Systems publication, and Dr. Anna Gagliardi, Senior Scientist at TGHRI, led the advisory group to redefine research assessment practices at UHN. (Photos: UHN StRIDe Team)

“To consolidate and prioritize these measures, we conducted two rounds of surveys with a panel of diverse researchers,” says Himani Boury, a previous practicum student at the Toronto General Hospital Research Institute and first author on the Health Research Policy and Systems publication.

The findings led to the identification of the following ten priority research assessment measures:


High Importance

Advancing existing knowledge

Innovative research plans

Independent funding support

Relevant research outputs

Publication content quality

Moderate Importance

Challenges to research productivity

Potential to improve health or health care

Team science

Collaboration

Recognition by professional/academic groups


“Our next focus was exploring the potential barriers to implementing and adopting these measures on an organizational level,” says Himani. “To do so, we conducted 18 interviews with researchers and research support staff from across UHN’s six research institutes.

“This shed light on a several strengths and barriers to implementation.”

Participants found the measures to be comprehensive, relevant across disciplines and rigorously developed. The reporting template for the measures was also deemed easy to understand and use. However, concerns were expressed over the measures’ relevancy across research disciplines. Another concern raised was that the complexity of the measures would make them more time-consuming to use.

To overcome these barriers, various strategies were suggested, including high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting tools, guidance for evaluators, and sharing of approaches among research institutes.

“Addressing barriers and implementing supportive strategies is crucial for the adoption of prioritized measures. These findings urge research organizations worldwide to reassess evaluation practices, embrace DORA-compliant measures, and work collaboratively to foster a transformative shift towards prioritizing quality and impact in research assessment,” concludes Dr. Gagliardi.

The work of the DORA Advisory Group will be continued by the Research Assessment Implementation Committee led by Drs. Michael Hoffman and Gagliardi.

Read more about this research.

This work was supported by UHN Foundation and The Princess Margaret Cancer Foundation.

DORA map
More than 23,000 individuals and organizations in 162 countries (coloured in yellow) across the world have signed the San Francisco Declaration on Research Assessment. (Source: sfdora.org)

This site uses cookies to provide you with a great user experience. By continuing on our site, you accept use of cookies. For more information, please visit our privacy policy.

CLOSE

Facebook Twitter LinkedIn Instagram YouTube