Algorithms
The scores per domain and the overall score for each of the rated trials can be easily calculated using the metapsyTools package in R. For this, reviewers should import the rating template (rob-template) in R/Rstudio.
Note: “No” also includes “PN” (probably No) response option, and “Yes” also includes “PY” (probably Yes) response option.
Domain 1
2 = No → High risk
2 = NI, and either 3 or 4 = No → High risk
2 = NI, and either 3 or 4 = Yes or NI → Some concerns
2 = Yes, 1 = No → Some concerns
2 = Yes, 1 = Yes or NI, and either 3 or 4 = No → Some concerns
2 = Yes, 1 = Yes or NI, and either 3 or 4 = Yes or NI → Low risk
Domain 2
5 AND 6 = Yes → Low risk
7 = NI or Yes, and 8= Yes → Low risk
7 = NI or Yes, 8 = No/NI, and 9 = Yes → Some concerns
7 = NI or Yes, 8 = No/NI, and 9 = No/NI → High risk
7 = No, any answer for 8/9 → High risk
Domain 3
10 = Yes → Low risk
10= No/NI, 11= Yes → Low risk
11= No/NI (analysis), 12= Yes (larger trial), 13= Yes (70% data), 14= Yes (missing unrelated) → Some concerns
11= No/NI (analysis), 12= Yes (larger trial), 13= No/NI (70% data), 14= Yes (missing unrelated) → Some concerns
11= No/NI (analysis), 12= Yes (larger trial), 13= Yes (70% data), 14= No/NI (missing unrelated) → Some concerns
11= No/NI (analysis), 12= Yes (larger trial), 13= No/NI (70% data), 14= No/NI (missing unrelated) → High risk
11= No/NI (analysis), 12= No/NI (smaller), 13= Yes (70% data), 14= Yes (missing unrelated) → Some concerns
11= No/NI (analysis), 12= No/NI (smaller), 13= No/NI (70% data), 14= Yes (missing unrelated) → High risk
11= No/NI (analysis), 12= No/NI (smaller), 13= Yes (70% data), 14= No/NI (missing unrelated) → High risk
11= No/NI (analysis), 12= No/NI (smaller), 13= No/NI (70% data), 14= No/NI (missing unrelated) → High risk
Domain 4
15= Yes, 16= Yes, 17= No/NI → Low risk
15= Yes, 16= No/NI, 17= Yes, 18= Yes → Low risk
15= NI, 16= Yes, 17= No/NI → Some concerns
15= NI, 16= No/NI, 17= Yes, 18= Yes → Some concerns
15= Yes/NI, 16= No/NI, 17= Yes, 18= No/NI → High risk
For meta-analyses including trials with multiple instruments/types of assessments:
15=No → High risk
15= Yes, 16= Yes, 17= Yes, 18= Yes → Low risk
15= Yes, 16= Yes, 17= No → Low risk
15= NI, 16= Yes, 17= Yes, 18= Yes → Some concerns
15= Yes/NI/No, 16= Yes, 17= Yes, 18= No/NI → High risk
Domain 5
19= NI/No → Some concerns
21 = No/NI → Some concerns
22= Yes → Low risk
22= NI → Some concerns
22= No → High risk
22= NI, 24= No → High risk
22= Yes, 24= No → High risk
Overall Score
- Low risk if all domains low risk.
- High risk if a trial has a high risk in at least one domain or multiple some concerns (so at least three domains with some concerns).
- Some concerns if at least one domain has some concerns and max. 2 domains with some concerns.
