The basic logic of evidence-based medicine is that researchers study health care interventions in rigorous clinical trials and then clinicians apply the subset of interventions with good outcomes to patients in the real world. However, this simple formulation glosses over the possibility that the research subjects in the clinical trial may differ from real-world patients in ways that make the results of questionable generalizability.
The primary reason this occurs is that clinical trials often forbid enrollment by many patients who are treated in our health care system, including for example anyone who is over the age of 60, or has multiple medical conditions, or is on medications etc. This makes the clinical trial easier to conduct but it can also result in a research sample that is completely unlike real-world health care recipients. If for example a new medication has been FDA-approved based on a clinical trial that excluded anyone who was already taking another medication, any adverse medication interactions won’t come to light until patients start experiencing them in the health care system.
This would be less of a problem if clinical trials carefully describe who was excluded from the study and why. To see how well researchers are doing on this score, my colleagues and I reviewed the 20 most highly-cited recent clinical trials for each of 14 prevalent medical conditions (i.e., 280 trials in all).
Our study appears in JAMA Internal Medicine today. The punchline: About half of even the most-highly cited clinical trials provide no information at all on how many patients were excluded and why. Among the subset that do reveal this critical information, the average study reports not enrolling about 40% of those patients with the disease who would get the treatment the study is evaluating in real-world clinical practice. By definition, these non-enrolled patients are different than those in the trial, because they were ruled out of participating by set criteria rather than at random.
You might wonder how we figure out whether the results of these clinical trials generalize to those who were turned away by the study eligibility criteria (or who were offered enrollment and refused). The general answer is that the treatments that performed well in the clinical trial are broadly provided in the health care system, and if someone notices bad outcomes accumulating among the type of patient who wasn’t in the original study, then we know the trial result has been over-generalized and what is beneficial to some patients is useless or harmful to others.
If that sounds a bit scary, it’s because it is.