Studying “Pre-Truth” Science: The 2023 Scholarly Communications Institute

Over the last four and a half years, I have become deeply obsessed with something I never would have expected: preprints.

If this term is unfamiliar to you, don’t worry: You’re not alone. When I started my PhD back in 2019, I, too, had no idea what a ‘preprint’ was. I started to understand while editing a series of (very technical) blog posts written by Mario Malički, Maria Janina Sarol, and Juan Pablo Alperin. But this was just the beginning of what would turn into a strange romance with preprints.

Through multiple rounds of frantic googling, I learned that preprints are papers that researchers make publicly available so that others can see and comment on their work. They differ from traditional academic articles, which are first submitted to a journal and are only published if they pass peer review. As a free way to make research publicly available, they are a core part of open science. They also let researchers get their results out into the world immediately—without having to wait months (or years) for their work to go through peer review and journal editing.

My true obsession began a few months later, when COVID-19 preprints started to make headlines. From research on how the virus spread to whether the controversial hydroxychloroquine drug was effective, un-peer-reviewed studies became a mainstay of pandemic media coverage.

These news stories fascinated and perplexed me. They raised so many questions that I couldn’t find answers to: Why do journalists use preprints? How do they vet these studies without the ‘safeguards’ of peer review? How do they make it clear to audiences that these papers are still unreviewed? Have preprints always been a part of journalism, or was the pandemic an exception? These questions became the basis of my dissertation—which I will defend just a few weeks from now.

But just as I thought my love affair with preprints was finally coming to a close, I was offered an opportunity to spend four days obsessing about them just a little more. With generous support from the Mellon Foundation—and an incredible team to keep me company—I made my way to Chapel Hill, NC, to attend the 2023 Scholarly Communications Institute (SCI).

This year’s institute brought together five teams of scholars and practitioners to work on projects broadly related to the theme of trust. Our team was led by François van Schalkwyk and included Yao-Hua Law, Mario Malički, Luisa Massarani, Chelsea Ratcliff, and myself. I was excited to collaborate with such a diverse group of journalists and researchers. Our project drew on our experiences working and living in six countries, as well as our shared expertise in scholarly communication, journalism, health communication, meta-research, and (of course) preprints.

Guidelines for Journalists: How to Report on Preprints

One of our team’s goals for the institute was to develop useful, evidence-based guidelines for reporting on preprints. We began by brainstorming guidelines based on our own research and experiences with preprints, peer review, scholarly communication, and journalism. We then compared this initial set of guidelines to existing recommendations for communicating about preprints. This helped us identify key tips that we might have missed and make sure we were adding something new—not just reinventing the wheel.

During our review, we noted that some existing recommendations were unrealistic for journalists working in fast-paced newsrooms or without deep expertise in science. We also found that guidelines focused solely on the work of journalists, and not on the editors, fact-checkers, and social media professionals who support (and sometimes control) their work.

With this in mind, we restructured our guidelines to reflect the multiple stages of news production: from generating the idea for a story through to publishing the final version. We plan to refine these working guidelines further by getting feedback from professional journalists, editors, and news professionals in the coming months.

Unanswered Questions and Future Directions: What Else Can We Learn About Preprints in Journalism?

During the institute, we spent a lot of time talking and brainstorming, both within our own team and with other SCI participants. Together, we explored questions about the state of science journalism, the (in)effectiveness of peer review, the future of preprints, the intersections of public engagement and scholarly communication, and more.

Some of these conversations were challenging, but they were also incredibly valuable. As Mario reflected, “It is so incredibly rare in today’s world that you are given an opportunity and support to spend several days in a row with a group of experts around the world to discuss a single topic.” François agreed: “In the hustle and bustle of academic life, the time afforded us was precious and necessary to advance our understanding of scholarly communication.” Luisa noted that the open-ended nature of the discussions was one of the most powerful aspects of the institute.

During our last day together, we reflected on these conversations and the many questions they had raised. We reviewed them as a team and created a list of those we felt were most important to explore in future research, including:

• Does explaining what preprints and peer review are make a difference for how public audiences understand preprint research?
• How does the public understand peer review? Do they trust research more if it has been peer reviewed?
• Are journalists using preprints to cover climate change? Can the patterns we saw during COVID-19 also be seen in the environmental media coverage of preprints?
• What do newsroom editorial policies say about preprints? Do these policies align with what their journalists or editors do in practice?
• What has changed since COVID-19, with respect to how journalists report on preprints?

With about 60 preprint servers, all of which describe and advise preprint use slightly differently, it’s clear that the field needs some unifying guidance. While our team plans to take on some of these questions to support that guidance, we also hope they inspire other scholars to join us. The use of preprints in journalism and scholarly communication continues to grow and change, and I know that our team will be along for the ride.