Journal Article Ahead of print 2 citations
PubMed: 41352579 DOI: 10.1016/j.jclinepi.2025.112097
Comparison between RoB-1 and RoB-2 tool and impact on network meta-analysis results - a case study from a living Cochrane review on psoriasis.
J Clin Epidemiol
Authors: R Guelimi, C Choudhary, C Ollivier, Q Beytout, Q Samaran, A Mubuangankusu, A Chaimani, E Sbidian, S Afach, L Le Cleach
Affiliations
- EpiDermE, Université Paris Est Créteil, Créteil, France; Department of Dermatology, Hôpital Universitaire Henri Mondor, Créteil, France.
- University College Hospital, London, United-Kingdom.
- EpiDermE, Université Paris Est Créteil, Créteil, France; Department of Dermatology, Hôpital Universitaire Henri Mondor, Créteil, France.
- Department of Dermatology, Hôpital Universitaire Henri Mondor, Créteil, France.
- EpiDermE, Université Paris Est Créteil, Créteil, France; Department of Dermatology, Montpellier University Hospital and Montpellier University, Montpellier, France.
- Université Paris Cité and Université Sorbonne Paris Nord, Inserm, INRAE, Centre for Research in Epidemiology and Statistics (CRESS), Paris, France.
- EpiDermE, Université Paris Est Créteil, Créteil, France.
Abstract
OBJECTIVES
This study was conducted within a large Cochrane living systematic review on psoriasis treatments with the aim to evaluate the inter-rater agreement of the Cochrane RoB-2 tool, to compare its risk of bias judgments with the original RoB-1, and to explore the impact of changes in risk of bias judgment between the two tools on the Cochrane network meta-analysis'(NMA) results.
STUDY DESIGN AND SETTING
This study was conducted within the 2025 update of a living Cochrane review on systemic treatments for psoriasis. Four pairs of assessors used RoB-2 to evaluate the risk of bias of 193 randomized controlled trials for two primary outcomes: Psoriasis Area Severity Index (PASI) 90 (reflecting clear or almost clear skin) and Serious Adverse Events (SAE). Inter-rater reliability (IRR) was calculated using Cohen's kappa. RoB-2 judgments for 147 trials (PASI 90) and 154 trials (SAE) were compared to the previous RoB-1 assessments from the Cochrane 2023 update. The impact of using RoB-2 versus RoB-1 judgments on the NMA's results was explored through sensitivity analyses, with calculation of ratio of risk ratios (RRR) between the analyses for each treatment effect.
RESULTS
For the RoB-2 overall judgment, the IRR was fair for PASI 90 (kappa = 0.37) and moderate for SAEs (kappa = 0.46). IRR varied between domains (from kappa = 0.33, to kappa = 0.65), with lower IRR found for domains 2, 3 and 5. Significant discrepancies were found between RoB-1 and RoB-2 judgments. Compared to RoB-1, RoB-2 rated a smaller proportion of results as low risk for both PASI 90 (36% vs. 58%) and SAEs (13% vs. 58%), and a higher proportion as high risk for SAEs (55% vs. 29%). For PASI 90, 66/147 (45%) studies showed switches between different judgments, including 18 extreme switches either from low to high, or from high to low risk of bias. For SAE, 93/154 (60%) studies underwent switches between different judgments, with 32 extreme switches occurring exclusively from low to high risk of bias. Sensitivity analyses excluding high-risk trials showed moderate impact on the NMA efficacy results (median RRR = 0.92, IQR 0.91-0.92), but wider changes for SAE (median RRR = 1.07, IQR 0.97-1.15).
CONCLUSION
The transition to RoB-2 in a large Cochrane systematic review revealed fair to moderate inter-rater agreement, underscoring the need for consensus among reviewers. The shift from RoB-1 to RoB-2 led to changes in risk-of-bias judgments in our review. While the impact on the NMA results was pronounced for SAE, the changes in results were limited for our efficacy outcome PASI 90.
MeSH
Bias, Epidemiology, Network Meta-Analysis, Therapeutics, Psoriasis