Building a Network of Research Transparency Ambassadors
Hello,
Our third – and best yet – US Biomedical Transparency Summit was held on January 24th in Washington D.C. A fabulous speaker and panellist line up, combined with the enthusiasm of over 100 transparency advocates convened at the beautiful National Academy of Sciences building, provided much cause for New Year optimism in achieving greater transparency in biomedical research.
As you know, the Center for Biomedical Research Transparency (CBMRT) hosts a free 5-year Biomedical Transparency Summit (BMTS) series in the US and Europe. The goal of BMTS is to promote discussion and collaboration on transparency across the global biomedical research ecosystem. If you couldn’t attend, read on to catch up on what you missed!
NIH and Reproducibility
Keynote speaker Michael Lauer, MD (Deputy Director for Extramural Research, National Institutes of Health) highlighted the longstanding and well-documented methodological problems inherent in scientific research that serve to negate reproducibility. These include an over-reliance on small sample sizes, failure to randomize samples and to blind study participants/staff.
“Unreliable research is inefficient and wasteful.”
Dr. Lauer outlined several promising initiatives to enhance methodological rigor and reproducibility. The NIH has a series of training modules that cover many aspects of rigor and reproducibility, including sex as a biological variable, research methods, and reviewer guidance. The NIH is also currently drafting its Policy for Data Management and Sharing which will require all NIH-funded grantees to submit plan outlining how researchers plan to manage scientific data, including when and where the scientific data will be preserved and shared.
“Initiatives that enhance methodological rigor are the best investment for public health.”
Transparency and Medical Journals
Howard Bauchner, MD (Editor in Chief, JAMA and the JAMA Network) noted an increasing willingness of authors to publicly correct their papers upon detection of errors. JAMA now publishes around 90 clarifications (letters to the editor) and 70 corrections each year. The ability to retract and replace papers was introduced in 2015 and there are presently around 4 retractions/replacements per annum. In the case of major errors, the journal has found that most of these result from coding mistakes which are usually detected by authors and result in large scale correction.
“To err is human, to correct divine. De-stigmatization of published errors supports accuracy and transparency.”
Dr. Bauchner also expressed concern over the impact of social media on scientific in an era of increasing open access and data sharing. He suggested that whilst data sharing has numerous benefits, it also comes with challenges, particularly in highly contentious and politicized areas such as the effects of climate change and pollution on health, in which social media discourse can sometimes stray from the facts, misrepresent underlying data, and become personalized.
“Virtually every time JAMA publishes an article on the effects of pollution or climate change on health, the journal immediately receives demands from critics to retract the article for various reasons.”
National Academies Work on Reproducibility & Replicability
Amanda Wagner-Gee (Program Officer, Health Sciences Policy) discussed three reports produced by the National Academies in recent years. Fostering Integrity In Research examines challenges to scientific integrity and recommended actions for all stakeholders. Open Science by Design provides guidance to the research enterprise in working to achieve open science, and Reproducibility and Replicability in Science is a study requested by Congress to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. The 2019 report also offers useful definitions of reproducibility and replicability.
Ms. Wagner-Gee also highlighted the National Academies workshop held late last year to focus specifically on enhancing scientific reproducibility in biomedical research through transparent reporting. In particular it focused on the possibility of improving the harmonization of guidelines across journals and funding agencies so that biomedical researchers propose and report data in a consistent manner.
A Strategy for Scientific Culture Change
Through its numerous initiatives, the Center for Open Science (COS) is making great strides in its mission to promote the openness, integrity, and reproducibility of research – Dr David Mellor (Director of Policy Initiatives) provided a useful progress update in the context of a ‘culture change pyramid’:
COS introduced its Transparency and Openness Promotion (TOP) guidelines for publishers in 2015; there are now over 1,000 journals implementing these guidelines (journals customize the eight modular standards, each with three levels of increasing stringency). Evaluating journal policies based on this
framework is now being conducted through the TOP Factor, a resource to rate and compare policies
across journals, fields, and publishers. COS is also seeing increasing journal uptake of its Open Science badges which aim to incentivize researchers to share data, materials, or to preregister – around 220 journals are now using the badges.
Also of interest is the role of registered reports in culture change. Dr Mellor cited a recent study which estimated the difference between the incidences of null findings in registered reports and that in the wider literature (a difference that can be interpreted as an estimate of the “dark data problem.”). Of the n = 296 hypotheses from registered reports analyzed, 60.5% were not supported by the experimental data; in stark contrast to around 12% of null findings in the wider traditional literature. By offering a path to publication of well-designed and adequately powered studies irrespective of null findings, the adoption of registered reports can help shift research culture towards greater transparency and rigor.
Driving Culture Change: The Null Hypothesis Initiative
Dr Sandra Petty (Co-Founder and CEO, Center for Biomedical Research Transparency) is on a mission to address the long-standing and well-documented issue of publication bias in biomedical research. CBMRT’s Null Hypothesis (H0) project involves partnering with major scholarly societies and journal publishers to publish special edition journals – titled H0 – in different therapeutic areas. H0 creates dedicated publication space for well-designed studies yielding negative, inconclusive and confirmatory results which are commonly perceived as more challenging to publish.
“Null Hypothesis enables researchers to just say it how it is.”
Dr Petty highlighted the success of CBMRT’s first H0 partnership (with Neurology® and its publisher Wolters Kluwer) over a reasonably short timeframe. CBMRT submitted its proposal to Neurology in mid-2017 which led to H0 being launched at the American Academy of Neurology Annual Meeting in 2018, and the first special edition Neurology® H0 being published in April 2019 (all papers appear online ahead of print and freely available on Neurology® website as they are accepted). CBMRT recently signed a MoU with Neurology® and Wolters Kluwer to continue publishing Neurology® H0 for another two years.
As noted by Dr Petty, H0 papers are useful, citable, of public interest and capable of informing research design and enhancing systematic review. CBMRT is expanding H0 into other therapeutic areas; Dr Petty encouraged BMTS participant organizations to nominate new areas for H0 partnership and to help fund innovative publication approaches such as H0.
“Negative results should not carry a negative stigma; they should be re-branded as valid and valued results.”
Journal Data Sharing Statements
Dr Bette Phimister (Deputy-Editor, New England Journal of Medicine) outlined journals’ ethical obligation to clinical trial participants and the broader research community to responsibly share clinical trial data. Journals’ commitment to data sharing has been formalized through the International Committee of Medical Journal Editors (ICMJE), which requires member journals reporting clinical trial results to include a data sharing statement (at time of manuscript submission from July 2018; and at the time of trial registration from January 2019).
“An infrastructure for making public data-sharing plans improves transparency and impacts authors’ actions. Authors of 50% of manuscripts reporting clinical trials intend to share at least some deidentified individual participant data*. However, at this point we don’t know the extent to which this is resulting in actual sharing of data.”
*Based on 263 NEJM manuscripts received between July 2018 and November 2019.
At present there is no enforcement mechanism; ICJME will continue to deliberate on mandating data sharing, bearing in mind the views of patients, trialists, editors, and funders and in the context of differing patient data sharing regulations across countries.
Preprints are Becoming Business as Usual. (Now what?)
Mr Todd Toler (VP Product Strategy & Partnerships, Wiley & Sons) presented a highly thought-provoking perspective on the evolution of preprints. He views research papers as living, evolving objects (there are about 400 events that take place in order to bring research into the public domain) and in this context assessing it at a single point of time may not be optimal. A preprint is one such event; preprints are increasingly being used to get early feedback and make associated revisions (over 70% of preprints posted each year are revisions). But Mr Toler noted also that researchers are increasingly using posters and abstracts at conferences (potentially instead of preprints) as the first point of results sharing and a means of generating early feedback.
“Publishers should not think of pre-prints as another siloed “content type.” Instead, they should be thought of as pre-publication sharing of works in progress.”
Mr Toler noted that publishers are increasingly integrating pre-prints into the research workflows, with 35 to 45% of authors now accepting the pre-printing option when presented as part of a journal submission process. This serves to tie preprints to journals as publishers deposit preprints on an author’s behalf after submission, either on journal-branded preprints or community preprints.
“We will soon get to the point where preprint is an option at every journal. And research funders are very interested in preprints; this turn will spur government research funders to action.”
Technology, Big Data and Transparency
Mobile health — the application of sensors, mobile apps, social media, and location-tracking technology to obtain data pertinent to wellness and disease diagnosis, prevention, and management is an exciting area of innovation in clinical care. Ida Sim, MD, PhD (Professor of Medicine, University of California San Francisco and Co-Founder of non-profits Open mHealth and Vivli) emphasized that to be useful, raw mobile health data must be processed into digital biomarkers (physiological and behavioral measures that explain, influence, or predict health-related outcomes; e.g. average nightly sleep duration) which can then be analyzed. And there are a number of challenges to address for the significant promise of mobile health to be realized:
Greater transparency and accountability, the setting of metadata standards (standards on how to describe data, such as their provenance), and external validation will all help to enable the benefits of mobile health application in clinical care to be realized.
Peer Review at PCORI
Dr Hal Sox (Director of Peer Review, Patient-Centered Outcomes Research Institute) described PCORI’s mandated peer review process for the research it funds, and its associated benefits and costs. From PCORI’s 2010 authorizing legislation (note that the first public funder to be required to publish comprehensive research reports, starting in 1997 was the UK National Institute of Health Research):
‘‘PCORI shall ensure that there is a process for peer review of primary research… Under such process… evidence from such primary research shall be reviewed to assess scientific integrity and adherence to methodological standards adopted… PCORI shall, not later than 90 days after the conduct or receipt of research findings… make such research findings available to clinicians, patients, and the general public.”
Since its inception in 2010, PCORI has completed 279 peer reviews of draft final research reports. It has posted 128 of these as final versions (the others are being prepared for posting). Whilst meeting these legislative requirements involves considerable expense and effort by both awardees and PCORI, Dr Sox highlighted the benefits of such public reporting: it promotes transparency and comprehensibility of research findings, helping both patients and clinicians in making healthcare decisions. Moreover, access to unpublished research can help researchers and funders to reduce waste in research.
Declaration On Research Assessment (DORA)
DORA aims to improve research assessment for hiring and funding decisions by reducing the reliance on journal-based metrics and capitalizing on technology to enhance evaluation of the impact of research. Dr Anna Hatch (Program Director for DORA) emphasized the community engagement approach being taken to successfully develop new research assessment practices and resources (over 1,800 organizations and 15,000 individuals are DORA signatories). It aims to engage researchers at all levels in discussions about assessment to clearly define concepts, set standards and curate new research assessment practices and resources.
Dr Hatch noted particular progress in the development of research assessment rubrics that include research integrity, legitimacy, and importance as part of the assessment. And structured narratives are also being developed to enable more even comparisons of candidates’ research track record. These narratives ask applicants not only about their contribution to a specific field, but also their contributions to open science, team science, and their interactions with stakeholders.
Thanks again to our fantastic speakers, panellists and participants for making BMTS’20 US a success. We’ve uploaded speakers’ presentations and summary videos to Figshare and we invite you to share this newsletter with interested colleagues/contacts so that we can continue building our network of ambassadors for research transparency. Finally, we’d appreciate your feedback and ideas on the newsletter (and CBMRT’s initiatives more broadly)… and don’t forget to follow us on twitter @CBMRT_org.
CBMRT, Inc, is a NY-registered 501(3)(c) non-profit organization. 175 Varick St NY NY 10014.
Please donate here to support our programs.
Click here to unsubscribe from this newsletter.
www.cbmrt.org
