Publication bias undermines the credibility of clinical trial reporting and negatively affects the reputation of pharma sponsors. Registered Reports (RR) is a peer review process designed to eradicate publication bias, enhance research publication quality and assist clinicians in appraising true drug efficacy and safety. In this article, we illustrate how RR could be harmonized with the US Food and Drug Administration (FDA) drug review process and leveraged to increase trust in pharma-funded research.
The problem of publication bias
Publication bias refers to the selective publication of research results depending on their strength and direction.1 Consequently, there is a bias in the literature towards the publication of positive trial results and away from negative studies.2,3 When negative results are suppressed, it prevents clinicians and researchers from being able to judge the true efficacy and safety of drugs and the differences between them. Publication bias has been identified throughout the peer-reviewed medical literature and the wider scientific literature.2,3
What is the RR process?
The RR process makes publication bias essentially impossible (see Figure), because the journal commits to publishing a paper based on the research question and planned methodology before the results are known.4
The RR format was initially offered by journals in the field of psychology, but it is now offered by more than 400 journals in multiple fields, including some of the Nature Portfolio journals. So far, medical journals have been slow to adopt RR, but we anticipate this changing once it becomes clear that RR can be harmonized with the FDA drug approval process.
Harmonizing RR with the FDA drug approval process
The flow diagram below summarizes how RR can be harmonized with FDA processes. Steps that go beyond the usual FDA review process are highlighted in red, specifically ‘Stage 1 peer review’ and ‘Stage 2 RR peer review’.
Figure. Registered Reports process flow overview
bSponsors are encouraged to share communications with the FDA publicly, but the final decision is at the sponsor’s discretion. Such communications are often included in FDA drug approval packages at a later date, and are routinely posted on Drugs@FDA.
FDA, US Food and Drug Administration; IND, Investigational New Drug Application; IPA, in-principle acceptance; NDA, New Drug Application; RR, Registered Reports; SAP, statistical analysis plan.
At Stage 1 peer review, without the study results potentially adding bias,5 the reviewer can focus on the scientific rationale and the methodological rigour of the manuscript, including the protocol and statistical analysis plan. Based on reviewer feedback, the manuscript is appropriately revised and granted in-principle acceptance (IPA). The sponsor then submits the study protocol to the FDA, which may conduct its own review, resulting in changes before the study begins. Any protocol revisions (and subsequent amendments) mandated by the FDA should be shared with the journal peer reviewers so that the IPA can be aligned.
The sponsor then conducts the study and, having collected the data, conducts the analyses according to the pre-specified methods. Next, the findings are written up and submitted for Stage 2 peer review. In contrast to the conventional peer review process, it does not matter if the results are statistically significant or deemed likely to change clinical practice; what matters is that there is no HARKing (i.e. hypothesizing after the results are known).6 The results are generated using the methods submitted and reviewed at Stage 1, and they are appropriately reported, emphasized and interpreted. Statistically
non-significant results for pre-specified primary outcomes therefore receive greater emphasis than significant results for secondary or post hoc outcomes.
Following successful Stage 2 RR peer review, the trial is published in RR format. In the interest of transparency, research sponsors are encouraged to share communications with the FDA publicly, but the final decision is at the sponsor’s discretion.
Limitations and obstacles
An initial barrier to leveraging RR for registration clinical trials is the limited number of medical journals that currently offer RR. However, we anticipate that once one or two top-tier medical journals introduce RR, others will quickly follow.
A potential concern among some pharma company stakeholders is that the RR process might mean having to report more negative/null results than their competitors, presenting a potential commercial disadvantage. However, as the RR format becomes more commonplace in the medical literature, so too will negative studies. In time, we believe that readers will increasingly understand that the conventional peer review process facilitates publication bias and will become increasingly sceptical of non-RR positive results.
Another concern may be that the two-stage RR process will take longer than conventional peer review. However, it is likely that any delay introduced by addressing early peer review comments at Stage 1 will mitigate delays and potential financial losses later in the process. Since 2021, RR can also be
peer-reviewed and recommended as preprints.
Finally, we should keep in mind that acceptance of a research plan by Stage 1 peer reviewers does not guarantee acceptance by the FDA. Change in a surrogate marker might be deemed a suitable primary outcome at early-stage peer review, but the FDA decides what outcomes are required for marketing approval.
Advantages of RR
By preventing publication bias, RR can help clinicians to distinguish between drugs more accurately according to their efficacy and safety, which allows them to make better-informed prescribing decisions. Empirical evidence has demonstrated that RR are associated with improved methodological and analytical rigour and with improved publication quality,7 which should increase article impact and trustworthiness, in addition to benefitting research funders who support the RR process.
Perhaps the greatest advantage to RR is that Stage 1 peer review allows the study sponsor to correct (potentially fatal) flaws in research design at a time when changes are still possible. By contrast, conventional peer review occurs after study completion, which disempowers peer reviewers who could otherwise help to refine the research methods. One example in which early peer review might have helped to address intrinsic study design limitations and mitigate downstream challenge is the case of functional unblinding in psychedelic drug trials (e.g. MDMA-assisted psychotherapy for post-traumatic stress disorder).8 If this issue had been properly addressed before the studies were conducted, the sponsor might have avoided an adverse FDA Advisory Committee vote and regulatory action,9,10 not to mention wasted years of work, millions of dollars and participant contributions.
Conclusions
In summary, we believe that RR could increase the transparency with which premarketing drug trials are reported, which would benefit research sponsors and public health.
Erick Turner MD is a former FDA medical reviewer and is Professor Emeritus of Psychiatry at Oregon Health and Science University. His research involves using FDA reviews to identify and quantify publication bias.
Chris Chambers PhD is Professor of Cognitive Neuroscience at Cardiff University. He is Co-founder of Registered Reports and other initiatives aimed at promoting and enhancing open research practices.
The views expressed in this blog post are those of the authors and do not necessarily reflect those of Open Pharma and its Members.
References
- DeVito NJ, Goldacre B. Catalogue of bias: publication bias. BMJ Evid Based Med 2019;24:53–4.
- Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics 2012;90:891–904.
- McGauran N, Wieseler B, Kreis J et al. Reporting bias in medical research – a narrative review. Trials 2010:11:37.
- Chambers CD, Tzavella L. The past, present and future of Registered Reports. Nat Hum Behav 2022;6:29–42.
- Mahoney MJ. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cogn Ther Res 1977;1:161–75.
- Kerr NL. HARKing: hypothesizing after the results are known. Pers Soc Psychol Rev 1998;2:
196–217. - Soderberg CK, Errington TM, Schiavone SR et al. Initial evidence of research quality of registered reports compared with the standard publishing model. Nat Hum Behav 2021;5: 990–7.
- Muthukumaraswamy SD, Forsyth A, Lumley T. Blinding and expectancy confounds in psychedelic randomized controlled trials. Expert Rev Clin Pharmacol 2021;14:1133–52.
- Lykos Therapeutics. News Release: Lykos Therapeutics Announces Complete Response Letter for Midomafetamine Capsules for PTSD. 2024. Available from: https://news.lykospbc.com/2024-08-09-Lykos-Therapeutics-Announces-Complete-Response-Letter-for-Midomafetamine-Capsules-for-PTSD (Accessed 14 November 2024).
- Kuperschmidt K. FDA rejected MDMA-assisted PTSD therapy. Other psychedelics firms intend to avoid that fate. 2024. Available from: https://www.science.org/content/article/fda-rejected-mdma-assisted-ptsd-therapy-other-psychedelics-firms-intend-avoid-fate (Accessed 14 November 2024).
