Domain 4: Measurement of the outcome
15. Is the method of measurement appropriate and applied using similar procedures for both groups?
Included instruments are valid and reliably measure the outcome domain of interest in the meta-analysis.
16. Were self-report measures used?
In the context of psychological interventions, blinding is often not feasible or even impossible to achieve. Empirical evidence shows that self-reports are not associated with overestimated treatment effects despite the lack of participant blinding (Cuijpers et al., 2010). Therefore, if there is no evidence that participants under-reported or there is a safeguard to prevent this (e.g., therapists did not have access to the self-reports), reviewers may consider self-reports as low risk of bias.
If it is likely that patients under-reported their severity (for example, to please the therapists or the researchers when self-reports are filled out in front of the therapist at the last session), reviewers should consider assessing item #16 as “No/PN”, which will lead to high risk of bias for domain 4.
17. Were assessor-rated instruments used?
Assessor-rated instruments are filled out by the therapist, independent evaluators, or research personnel. These are usually administered as a structured interview with the participant or are completed by the evaluator through observation of the participant’s behavior.
18. If yes to #17, were the assessors masked to treatment allocation?
This item evaluates whether the assessors evaluating the outcome were masked to treatment allocation. This item should be rated as “Yes” when trialists clearly report that the assessors were masked. If this is not reported, it might be sensible to assume that no masking procedures were implemented (“No/PN”).
Meta-analysts might be interested in including all available results for one outcome domain within a trial. For example, in a depression trial, there might be post-test data from the Hamilton Depression Rating Scale (assessor-rated) and the Beck Depression Inventory (self-report). There could be two strategies for these cases, depending on the meta-analysis protocol and analysis approach:
- Separate risk of bias scores could be performed for each outcome and numeric result, or
- Two or more ratings could be combined into an aggregated score, conservatively rating the aggregated score as high risk when one of the measurements is at high risk.
For the Metapsy depression database we use the second strategy, combining ratings into an aggregated score.
