Domain 5: Selection of the reported result
Similarly to the previous domain, the evaluation of this domain will depend on the meta-analysts’ protocol regarding the choice of outcome measures. Some meta-analysts might decide to include all available measurements for one outcome domain (e.g., depression symptoms) (even if this entails including multiple measurements from within a study), while others might decide on more restrictive approaches (e.g., including only assessor-rated depression severity). To appropriately evaluate the selection of the reported result, reviewers should take into account the meta-analysis protocol for inclusion of outcome measures and results.
19. Is there a trial registration, protocol, or statistical analysis plan (SAP) available?
If there is no access to documents describing the trial’s protocol pre-defined outcome measures or analysis plan reviewers should answer “NI” in this item. The trial can then be rated as “Some concerns” for domain 5 and the ratings stop here.
If reviewers have access to such documents, the next questions should be answered.
20. Please write down the registration number, protocol, or link to SAP.
21. If “Yes/PY” to #19, were outcomes (scale and time point) pre-specified before unmasked outcome data were available?
This item evaluates whether the outcome measures in a trial were pre-specified before the start of data collection (i.e., within one month of the start of participant enrolment). For example, when the trial was registered (and specified the outcomes) before the start of data collection, which is known as prospective registration. To evaluate whether registration of outcomes was prospective, reviewers can compare the date in which recruitment started and the date in which the registration was posted (e.g., in ClinicalTrials.gov). This differs per trial registry, as some registries already give the information of whether the registration was prospective (e.g., ISRCTN Registry). It is important to also examine the “history of changes” that is available in some of the registries, as unjustified changes throughout the trial after unmasked data was available could have occurred.
Outcome domain, scale, time point, metric, and analysis should ideally be specified, although reviewers can answer this item as “Yes” if at least scale and time point were prospectively specified.
Retrospective registrations (registering the trial after start of data collection) and lack of information (no scale/time specified) in a prospective registration should lead to “NI” (respectively) in this item, both resulting in “Some concerns” in domain 5.
22. If “Yes/PY” to #21, are all outcomes of interest for the meta-analysis fully reported in the paper or available through other means?
This will partly depend on the meta-analysis protocol. If meta-analysts aim to include all available instruments for a given outcome (e.g., depression severity), the article should report all instruments in full (with enough data to be entered in the meta-analysis). If meta-analysts have a pre-specified hierarchy for instrument inclusion (e.g., select HAM-D over BDI), then selective reporting will be judged regarding the availability of outcomes based on the pre-specified hierarchy. It will also be rated as high risk when a non-registered outcome is added in the publication of the paper. Reviewers should answer this question with “No/PN” in this case. Reviewers should consider whether data from the outcome of interest can be made available through other means (e.g., contact through authors).
23. If “Yes/PY” to #19, Was there an analysis plan pre-specified?
Pre-specified plan = before unblinded outcome data is available.
Dates of publication of the analysis plan should precede participant enrollment. Published protocols: Acceptance date of the published protocol precedes participant enrollment. Reviewers should answer with “NI” if it is not clear whether the protocol reflects a pre-specified plan.
24. If “Yes/PY” to #23, Was data analyzed in accordance with the pre-specified plan?
Any deviation from the original plan should be reported in the final publication and should be justified. If properly justified, meta-analysts can rate #24 as low risk, after careful assessment of how this deviation can affect the studied meta-analytic result.
We will not consider item #24 in the algorithm for domain 5 unless there is evidence of unjustified deviations from a pre-specified analysis plan, in which case the meta-analysts should consider downgrading the domain to high risk when necessary.
