How to generalize efficacy results of randomized trials: recommendations based on a systematic review of possible approaches

How to generalize efficacy results of randomized trials: recommendations based on a systematic review of possible approaches:

Abstract

Rationale, aims and objectives

Randomized controlled trials (RCTs) are the preferred source for evidence for the effect of treatment. However, patients participating in RCTs often manifest important differences from patients seen in practice. Therefore, guideline developers have to decide whether the results are generalizable to the target population not represented in RCTs.

Method

A systematic review of the literature was undertaken to identify methods to decide whether to generalize the results from RCTs to patients who were not represented in these trials.

Results

One approach is to examine the in- and exclusion criteria of trials and infer from these whether the trial population was sufficiently representative. Other authors suggest, because of the inclusion of a broader range of patients, reliance on observational studies if no direct evidence for the target population is available.

Another approach is to apply the relative effect of treatment found in trials to patients in practice unless there is a compelling reason to believe the results would differ substantially as a function of particular characteristics of those patients. Although there are exceptions, this approach is supported by empirical evidence that, in general, relative effect of treatment on benefit outcomes seldom differs to an important extent across subgroups of patients.

Conclusion

We propose this last approach: focusing on RCTs unless there is a compelling reason not to do so. Compelling reasons will most often be found with respect to issues of rare adverse effects, for which observational studies are likely to provide the best estimates.

abstract: Assumptions of Expected Benefits in Randomized Phase III Trials Evaluating Systemic Treatments for Cancer

Assumptions of Expected Benefits in Randomized Phase III Trials Evaluating Systemic Treatments for Cancer

Background In designing phase III randomized clinical trials (RCTs), the expected magnitude of the benefit of the experimental therapy (δ) determines the number of patients required and the number of person-years of follow-up. We conducted a systematic review to evaluate how reliably δ approximates the observed benefit (B) in RCTs that evaluated cancer treatment. 

Conclusions Investigators consistently make overly optimistic assumptions regarding treatment benefits when designing RCTs. Attempts to reduce the number of negative RCTs should focus on more realistic estimations of δ. Increased use of interim analyses, certain adaptive trial designs, and better biological characterization of patients are potential ways of mitigating this problem.

Editorial :: VIEWPOINT: Randomised controlled trials using invasive ‘placebo’ controls are unethical and should be excluded from Cochrane Reviews - The Cochrane Library

"Placebo controls are frequently used to ‘blind’ participants, trial personnel and outcome assessors to intervention and control in clinical trials. Effective blinding of treatment reduces the risk of performance bias (differences between groups in the care provided apart from the intervention being evaluated) and detection bias (differences between groups in how outcomes are ascertained, diagnosed or verified). A placebo has traditionally been defined as an “inert or innocuous substance”,[1] such as a ‘sugar pill’. However, some randomised controlled trials (RCTs) have been shown to erroneously use the term ‘placebo’ to describe an invasive intervention that exposes participants, allocated to a control group, to risks of serious harm.[2,3] In this context therefore, the term ‘placebo’ describes an invasive intervention which is neither inert nor innocuous.
In a recent study of local anaesthesia RCTs, over half the RCTs used an invasive ‘placebo’ control.[2] The ‘placebo’ interventions mostly involved deep-needle insertion through body tissues with potential damage to nerves, vessels and other structures such as liver and bowel. These interventions exposed control group participants to risks of serious morbidity.[2,3].......

PLoS Medicine: Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials as Compared to Observational Studies: Methodological Overview

Note: PLOS Medicine is an open-text publisher (free full access), below are a couple of excerpts of interest

.....................................................

Background

There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.

Discussion Top

"............The increased risk in adverse effects in some studies was not consistently related to any particular study design—RCTs found a significant risk of adverse effects associated with the intervention under investigation in eight instances, while observational studies showed a significantly elevated risk in 11 cases.
............Although reasons for discrepancies are unclear, specific factors which may have led to differences in adverse effect estimates were discussed by the respective authors.................
..........While there are a few instances of sizeable discrepancies, the pooled estimates in Figure 2 and Table 2 indicate that in the scheme of things (particularly where larger, more precise primary studies are available), meta-analysis of observational studies yield adverse effects estimates that broadly match those from meta-analysis of RCTs..........."