John Ioannidis, M.D., DSc., the C.F. Rehnborg Chair in Disease Prevention at Stanford University and co-director of the Meta-Research Innovation Center at Stanford, addressed challenges with optimizing research practices in medicine Tuesday afternoon at the 2018 McGovern Medical School Research Retreat.
Ioannidis opened by focusing on the massive number of research papers published worldwide and how the frequency of those reporting “statistically significant results” has rendered many of those results and discoveries a “boring nuisance.”
“Everything you read sounds extremely important,” Ioannidis said.
However, a study found that 96 percent of online medical literature claim to have statistically significant results, something Ioannidis found to be too good to be true. He said 40 percent of such papers are refuted within a few years. While there are exceptions, those “expedited success stories” such as triple therapy for HIV entering clinical practice in a manner of years, it takes a long time to translate such discoveries into something that can affect major clinical outcomes. On average, it takes about 25 years for such a process to take place.
Ioannidis said over the last few years there has been an epidemic of reproducibility literature, as many different scientific fields see a growth of the use of such terminology. The challenge comes from the different meaning behind reproducibility, including reproducibility of methods, of results, and of inferences.
“Many of the recipes of research practices we use are not conductive to getting reproducible results,” Ioannidis said. He separated these practices into two major sets – those practices with big data and those with small data. With small data, acquiring and analyzing new information has to be funded with grants, and funding is limited across investigators who must renew such grants consistently. However, to lobby for these grants, investigators have to say they have found something major.
“This means, by default, cherry picking of the best-looking results,” Ioannidis said.
For research practices focused on big data, researchers face similar challenges, but the problem now moves to the opposite end of the spectrum. In some of the electronic health record databases there are tens of millions of participants’ worth of data, which Ioannidis said that almost any association a researcher may want to test would be “statistically significant.”
“Now the problem is not that we’re underpowered, but that we’re overpowered,” Ioannidis said.
Empirical studies on fields where replication practices are common suggests that most of the initially claimed statistically significant effects are false positives or substantially exaggerated, Ioannidis said. He suggested different interventions that can safeguard the design, analyses, and reporting of results in papers, particularly at the earliest stages of a study.
“I think that any amount of thought that goes into thinking about the study early on is not wasted,” Ioannidis said. “The more effort I invest early on, the better.”
Ioannidis suggested some research practices that would help increase the proportion of useful findings, such as greater sharing of data, adopting reproducibility practices, the standardization of definitions and analyses, and more stringent thresholds for claiming discoveries or “successes.”
“All of these practices depend on whether we incentivize them,” Ioannidis said. “If institutions, funders, or journals show that they care about them, they will happen.”
Ioannidis concluded the keynote by asserting the reproducibility and usefulness of many disciplines of biomedical research and beyond has substantial room for improvement. He said there are many possible interventions that may improve the efficiency of research practices and empirical meta-research would be useful not only to assess the prevalence of problems but also the effectiveness and potential harms of interventions that try to make research more reproducible and, eventually, more useful.
Ioannidis also said the reward system for researchers could be re-engineered.
“Until now, we are honoring productivity,” Ioannidis said. “I have nothing against productivity, […] but I think we need to look at other dimensions as well. We need to find ways to reward quality, reproducibility, sharing of data, and the translational impact of research.”