Hello fellow journalologists,
The received wisdom among editors is that the best way to boost a journal’s impact factor (JIF) is to reduce the number of citable items that the journal publishes. But is this true?
This question is especially important right now because many publishers are asking their editors to increase the number of articles they publish to help mitigate against the financial risks of the open access transition.
(Publishers, commercial and not-for-profit, often are only able to make half as much revenue from an open access (OA) article processing charge (APC) as they do under a subscription model.)
There’s increasing tension between editors and publishers as a result; editors want to maintain quality (i.e. selectivity) whereas publishers want their journals to increase article volumes.
A former colleague of mine, Pep Pàmies, the Chief Editor of Nature Biomedical Engineering, wrote a typically insightful editorial on how article volume and impact factors are correlated over time. Pep introduces his editorial as follows:
Yet because the distribution of citations approximately follows a power law with a long tail of increasingly fewer papers each capturing more citations, and because increasing the quality of submissions is a protracted and laborious task, editors rightly expect that they can boost the journal’s IF earlier by increasing the journal’s selectivity — that is, by publishing fewer of the low-cited manuscripts and hence increasing the relative weight of the distribution’s long tail. Are they right?
To try to answer this question, Pep plotted graphs that showed the year-on-year change in JIF against the year-on-year change in citable items. This creates four quadrants: dots that appear in the top right of the graph represent years where both the citable items and impact factor for a given journal increased; by contrast, dots that appear in the top left represent years where the number of citable items decreased and the JIF increased.
Yet largely the opposite has occurred in the past 10 years: for most journals and most years, the figure shows that more papers published are followed by increases in IF, and that substantial reductions in citable output (that are either deliberate or because of reductions in submissions owing to strong competition) may not lead to a boost in IF and can even cause it to drop.
This phenomenon is counterintuitive because JIFs are a ratio of citations to the previous 2 years’ worth of content. For example, if a journal published 100 articles in 2020 and 200 articles in 2021 then its total citable item count, for the 2022 impact factor calculation, would be 300.
Papers published in the second year (2021; n=200) are less likely to accumulate citations (in the 2022 JCR) as papers published in the first year (2020; n=100). Why? Because papers published in December 2021 are less likely to be cited in 2022 than papers published in January 2020. In other words, rapid growth should lower impact factors.
But that’s not what Pep observed, so what's going on? Pep provides some possible explanations in his editorial. My hunch (and it is just a hunch) is that the more papers a journal publishes, the more authors become aware of the journal. This increased visibility may increase the likelihood that the author cites the journal, which increases the number of citations (and hence the JIF). This, in turn, results in more submissions. In other words, it’s a perfect example of Jim Collins’ flywheel effect.
This may be why many journals have been able to increase, or at least maintain, their JIFs while growing rapidly. It’s worth remembering that some journals have increased the number of review articles that they’ve published in recent years, which likely helped to boost their JIFs too, since reviews tend to be cited more than original research. This is especially true of the Frontiers journals, which publish large numbers of Review articles. For example, Frontiers in Immunology, which has a JIF of 7.3, published 8,416 articles in 2022, of which 2140 were Reviews (source: PubMed; filtered to only include articles with abstracts).
Another factor that could explain why JIFs often increase in tandem with citable items is impact factor inflation; Phil Davis’ 2010 post in the The Scholarly Kitchen gives a good explanation of that effect. Indeed, Pep cites a 2009 paper, which suggests that JIFs have grown over time, across the industry, because reference lists have tended to get longer. If you want to use Pep's technique to assess your portfolio, it would probably make sense to normalise for JIF inflation in some way.
I suspect (but have no evidence) that portfolios that have their brand as a prefix in the journal name benefit most from flywheel effects. Perhaps that’s why brands such as Nature, Lancet, JAMA, JACC, Physical Review, Frontiers etc. have become so dominant in recent years — researchers are submitting to them in increasing numbers because brand recognition has been boosted by higher article volumes (MDPI is the exception that proves the rule, of course). Branded portfolios are appearing with increasing frequency at the top of the JCR. Success breeds success.
Flywheels can work in reverse too and the big challenge for rapidly growing journals is to ensure that turnaround times don’t increase (i.e. become slower). Popular journals that are inundated with submissions can struggle to keep up. Slow journals become unpopular journals, which can put a brake on a flywheel. PLOS One and Scientific Reports both experienced this problem in the past. Will the same happen to Frontiers and MDPI?
I’d be interested to hear your thoughts on this hypothesis. Please do hit [reply] to tell me if you agree or disagree with what I’ve written here. This newsletter is a mechanism for me to “think out loud” ━ please don’t take what I’ve written here as gospel.
News
The papers’ peer reviews, posted on 12 July, come to much the same conclusion about the scientific evidence. After citing a litany of missing evidence, one reviewer wrote: “The manuscript in its current condition is deemed incomplete and inadequate, and should not be viewed as finalized scholarship.” Berger says that his team is still taking in the reviews, and that the group plans to address some — but maybe not all — of the concerns in future versions. “We haven’t published our final paper yet.” He says that the team will stop seeking further review “when we feel that we have come as close to meeting the valid criticisms as we could”.
Nature (Ewan Callaway)
JB: It’s worth reading the eLife editors’ assessment of this paper (which was published at the same time as a high profile Netflix documentary) and the peer review reports, which were damning: “The four reviewers were in strong consensus that the methods, data, and analyses do not support the primary conclusions.”
Under the eLife model, the authors get to choose which of the referees’ comments to address before publishing the final version of their paper. eLife publishes all articles that are peer reviewed, even if the referees trash the paper. Nature’s news story suggests that the authors may not be willing (or able) to address the peer reviewers’ concerns.
In a previous newsletter I noted that eLife’s new publishing model is an interesting (if rather risky) experiment. For me the big question is whether Clarivate will delist eLife from the JCR because it essentially removes the editorial filter; there is no ‘reject’ decision in the new model. If the journal loses its (very respectable) impact factor, will submissions rapidly decline?
Incidentally, the handling editor deserves lots of credit for explaining the editorial process in this Twitter thread. I wish more editors would be as open and transparent about their decision making as this.
A prominent journal has decided to retract a paper by Ranga Dias, a physicist at the University of Rochester in New York who has made controversial claims about discovering room-temperature superconductors — materials that would not require any cooling to conduct electricity with zero resistance. The forthcoming retraction, of a paper published by Physical Review Letters (PRL) in 20211, is significant because the Nature news team has learnt that it is the result of an investigation that found apparent data fabrication.
Nature (Dan Garisto)
JB: Twitter (sorry, X) went into meltdown about a new paper, from a different research group, entitled: The First Room-Temperature Ambient-Pressure Superconductor. It’s a preprint and hasn’t been peer reviewed. Hopefully this one hasn’t been fabricated.
Springer Nature, the world’s leading publisher of protocols, has acquired protocols.io - a secure platform for developing and sharing reproducible methods… protocols.io will form part of Springer Nature’s expanding Solutions business which is committed to providing researchers, and their institutions, with a comprehensive suite of tools and services designed to bolster their success, enhance their impact, and boost productivity.
Springer Nature Group (press release)
Since we started work on building protocols.io in 2012, it has been a non-stop challenge to secure sufficient funding. In spite of limited resources, we succeeded in creating a terrific tool for researchers and have a vibrant community of scientists that use it for both private collaboration and open sharing. Our mission has always been to secure and ensure long-term stability for the platform so that we can continue to deliver for our research community. The acquisition by Springer Nature brings us to this next level with proper resources and stability. We retain our independence as a business unit, but will sit within Springer Nature where we will be able to leverage their scale and position as a leader in the protocol-sharing space - expanding our collective reach to focus on our shared goals of serving the research community. Our current team will continue their amazing work with the additional resources that come from being a part of a larger global publisher. That includes doubling our engineering team and hiring for other roles to enable us to better focus our energies on serving the community and growing and improving protocols.io.
protocols.io news (Lenny Teytelman)
JB: An ‘independent business unit’, eh? Is there such a thing in a large corporation like Springer Nature? Look for an assessment of this acquisition in the next issue of The Brief, which will arrive in your inboxes very soon.
This COP [Community of Practice] will provide the opportunity to hear and learn from each other, share resources and best practices, hear of challenges and successes, explore opportunities for participating in cross-industry initiatives, coordinate effort, and to speak with one voice to increase effectiveness.
Coalition for Diversity and Inclusion in Scholarly Communications
Many researchers worried by medical fakery agree with Carlisle that it would help if journals routinely asked authors to share their IPD [individual patient data]. “Asking for raw data would be a good policy. The default position has just been to trust the study, but we’ve been operating from quite a naive position,” says Wilkinson. That advice, however, runs counter to current practice at most medical journals.
Nature (Richard van Noorden)
Early this year, Sage became aware of concerns with the journal’s content after one article “showed indicators of third party activity and compromised peer review,” a spokesperson said. The publisher began an investigation and reviewed “all submissions that showed indicators of suspicious activity. We asked which article tipped off the investigation, but Sage was not able to share the title by our deadline.
Retraction Watch (Ellie Kincaid)
In partnership with our community members, over the next six months Dryad will be piloting a more equitable and sustainable institutional membership model that also allows us to support growing demand and safeguard community ownership of Dryad. We are also standardizing our publisher membership model to align with our values of trustworthiness and inclusion.
Dryad (announcement)
eLife and PREreview are pleased to announce that the Confederation of Open Access Repositories (COAR) will provide them with technical and funding support to implement the COAR Notify technology. With this support, the organisations will work to connect separate services within the ‘publish, review, curate’ ecosystem. The project will put in place the basic infrastructure and protocols needed for all-round and standardised connections between preprint repositories, community-led preprint review platforms, journals, and preprint review aggregation and curation platforms. The aim is to lower existing technological and cost barriers so that as many of these services as possible can more easily participate in the ‘publish, review, curate’ future for research.
eLife (press release)
But an individual preprint can be revised, commented on, and peer-reviewed, and a version can eventually be published in a journal. To help you keep track, medRxiv is adding a new feature called
Follow a Preprint
which notifies you when any of these events occur.
medRxiv (announcement)
Community Corner
Daniel Muzio, who is Professor of Management at the University of York and is also General Editor at the Journal of Management Studies, contacted me recently to discuss The Organization and Management Editors’ Network (OMEN). The OMEN website went live this week and lists three aims for the organisation:
- Provide a networking platform where editors can discuss issues of common concern
- Encourage capacity building by offering training events for editors
- Provide a vehicle to comment on issues of common concern and influence developments in our field
Self-help groups like this one are incredibly important, especially for academics who are juggling their editorial work alongside their research, teaching, and (for medical journal editors) clinical commitments.
Daniel was kind enough to leave a testimonial on the Journalology Community Wall. Thank you, Daniel, and good luck with OMEN.
Opinion
In scholarly publishing, the incumbents are subscription-based publishers. And the benefits of incumbency are clear. The worldwide scientific publishing market for journals is around US$ 27 billion. The five largest paywall publishing houses (Elsevier, Wiley, Springer Nature, Taylor & Francis, and SAGE) have captured more than half of it.
The Scholarly Kitchen (Tom Ciavarella)
JB: I very nearly left my first-ever comment on a TSK post because the $27 billion statistic quoted in the blog is wrong. Paul Abrahams, the Chief Communications Officer at Elsevier, beat me to it. I’ve seen this mistake made a number of times. According to the STM report quoted, the Journals market was $9.5 billion in 2020, not $27 billion. The TSK editor pointed this mistake out pre-publication, but the author chose not to correct it. It’s still incorrect on the TSK website.
You can read more about the ‘attack on the Nelson Memo’ in the next issue of The Brief. Some open access advocates have jumped to the conclusion that the evil commercial publishers are behind this ‘attack’, but it looks as though the truth is far more unsavoury than that.
The OSTP’s memo will be especially impactful in the realm of data sharing, as it asks federal agencies to develop plans to require data underlying published studies to be shared immediately upon publication and explore strategies for sharing all data, even if not tied to a published study. This policy advance takes place within the larger context of international efforts to enable open data sharing at scale, including the national open science plans in Ireland, Colombia, Spain, and France. As contributors to the Higher Education Leadership Initiative for Open Scholarship and the Open Research Funders Group, we believe this moment in time represents an unparalleled opportunity to elevate research software as a core component of the scientific endeavor and to take specific steps to ensure its open and equitable availability.
PLOS Biology (Erin C. McKiernan et al)
As recognition in peer review is yet to be applied across the sector, I would like to see ghost-written peer review reports become a thing of the past. Giving every reviewer credit for their contributions to a review report enhances transparency and gives credit where credit is due. Introducing co-reviewing fosters collaboration, alleviates reviewer pressure, and supports a more inclusive future of peer review. By implementing co-review and opportunities to give feedback to reviewers, publishers can strengthen the peer review process and contribute to the advancement of scientific knowledge and innovation.
The Scholarly Kitchen (Laura Feetham)
2023’s Global Goals Week is taking place during September 15-24. The key purpose of this campaign is accelerated action, improved awareness, and stronger accountability for the SDGs. While the world will be taking stock of the SDG progress in September, what are scholarly publishers and their associations going to do? Will some of us come together and issue a joint statement as we did ahead of the climate change conference (COP26) in 2021? Or will we go beyond that? Who is going to take lead, and facilitate accountability across the publishing ecosystem?
The Scholarly Kitchen (Haseeb Irfanullah)
In 2021, the news agency Reuters released its list of 1,000 top climate scientists. It included only one woman in the top 20 and only 7 in the top 100. Authors from lower-income countries were barely represented. It was ridiculous. But, along with several other editors at the journal Climate and Development, we published an editorial response around three weeks later, highlighting steps that scholars, editors and publishers could take to close the inequality gap between the global north and global south.
Nature (Minal Pathak)
Scientists should make a more concerted effort to share their physical samples. Doing so would improve reproducibility, spur innovation, and help level the playing field in a pursuit where funding and resources are unevenly distributed… Any one-size-fits-all solution is unlikely to work. To cram paleontology, virology, materials science, and science’s myriad other disciplines into the same sample-sharing box would be foolish. But the notion of a world in which scientists work more openly and collaboratively to benefit society is not. Though it may sound unfeasible, even crazy to some, we think that world is within our reach.
Undark (Tim Verhagen and Julie Nováková)
English-language dominance in publishing reinforces barriers for scholars whose first language is not English. These barriers impact peer review: authors whose primary language is not English experience worse review outcomes. This is probably due to reviewer and editor bias: authors whose first language is English are more likely to be favourably reviewed and invited for resubmission, but this difference disappears when author identity is blinded to reviewers. These empirical data shed an uncomfortable light on the academic publishing industry. We strive for greater diversity, equity and inclusion, yet we continue to cause minoritized scholars to experience injustice.
Nature Human Behaviour (unsigned editorial)
The coming shift — which is admittedly already underway in certain sectors — will recentre dissemination from distribution to discovery. Efforts by publishers to syndicate content to ResearchGate and ScienceDirect are harbingers of the coming focus on discovery. It can no longer be assumed that a work will find its audience — even works that are published open access. Instead, publishers will have to shift resources from production to ensuring discovery of what is published, investing in robust metadata, content syndication and expanded programmes of use tracking and analytics. The danger here is that discovery efforts will tip into hype-marketing, although any journal that does so will probably risk its reputation for intellectual integrity. Through a focus on discovery, scholarly journals should be able to counter the pressures that lead some to believe their demise is imminent and to reestablish their role as a central actor in scientific communication.
Nature Human Behaviour (Lisa Janicke Hinchliffe)
JB: The editors of Nature Human Behaviour have done a great job of bringing together a diverse set of viewpoints. I chose to quote Lisa because her opinion tallies with my view of how things will play out. Publishing a paper is not enough. It has to be read by its target audience too.
Augmenting a research article with publicly available code enhances understanding, facilitates reproducibility and reanalysis, promotes trust, and saves other researchers time and effort. Interestingly, code generation does not appear to have a strong relationship with code sharing. In topic areas with high rates of code generation, code-sharing rates were widely distributed. For example, Mathematics, Information and Computing Sciences, and the Physical Sciences exhibit higher rates of both code generation and sharing, while the Biological Sciences, Environmental Sciences, and Chemical Sciences had equally high rates of code generation, but much lower rates of code sharing.
The Official PLOS Blog (Lauren Cadwallader and Lindsay Morton)
But why do we want this “forkability”? The simple answer, as with many things at Crossref, where I work, is: persistence. The scholarly infrastructures on which we depend for (say) linking and metadata resolution must continue to exist and there must be mechanisms for their ongoing operation. POSI is designed to allow an arbitrary third-party entity to continue the operation of an infrastructure in the event of the original organization’s demise – or in cases where the original organization might be acting against the wishes of the community.
Martin Paul Eve blog
Our previous analysis in June 2021 suggested that journals in the TJ program had historically grown their OA proportions at half the rate they needed to. We may have been slightly pessimistic, but not by much. There has been no significant leap in OA proportions. Over two thirds of TJs have already lost their status, and well under one fifth will make it to the end of the program.
If the real point of the TJs was to encourage flips to fully OA journals, just 3% have flipped (or are due to flip in 2023), and fewer than 30% of TJs are likely to make the 75% OA threshold by 2028. The program appears to have had little effect on OA growth, and cOAlition S sees this slow uptake of OA as justification to close the program (end the experiment?) at the end of 2024 as planned.
Delta Think (Dan Pollock and Heather Staines)
Wiley is a mighty, major international publisher, and they have the potential to achieve a Hindawi revitalization, if revitalization is defined as a significant rebound in the number of papers published in Hindawi journals. The question is, which niche market does Hindawi intend to regain? Are they well-positioned to do so? Do they have any appreciation of the tension between their goal of publishing as much as possible, and the reputational costs of publishing papers that are low-quality at best and fraudulent at worst?
BishopBlog (Huanzi Zhang)
JB: I think we can safely assume that Wiley is very aware of this tension…
One of our roles has always been to demonstrate what’s possible and so that’s again where our focus is now shifting. We have a vision for a radical reframing of research sharing, built on the principles of Open Science, based on rigor, openness, and equity.
That vision is something we’re now making concrete as we create a new way of sharing research. This will shift us away from the final article being the sole center of attention to one of a series of research objects, shared and appropriately assessed at different points in the research lifecycle. One that demonstrates that the article doesn’t have to be immutable but that conclusions may shift over time. And one in which peer review evolves into less of a binary accept/reject decision to a more nuanced assessment.
The Official PLOS Blog (Alison Mudditt)
As an editorial team, we bear the responsibility of shaping narratives, influencing perspectives, and representing the voices of our diverse readership. It is imperative that all those involved in the editorial process acknowledge and champion the principles of diversity, equity, and inclusion (DEI). These principles transcend mere buzzwords and form the foundation for progress, empathy, and empowerment of diverse communities. It is our duty to embrace and amplify diverse voices, ensuring our work resonates with all readers and contributes to building a more inclusive society. In line with these principles, we will be establishing an Early Career Editorial Board, comprising emerging researchers who have not previously served as Assistant/Associate Editors of a journal but possess the drive to shape the journal's content. If you are interested in contributing to the journal in this capacity, please reach out to me.
The FASEB Journal (Loren E. Wold)
JB: It’s always interesting to read a recently appointed Editor-in-Chief’s first editorial, and this is a good one. More journals should have early career researchers as part of their editorial boards.
In some cases, the request for further work/data is certainly relevant, and useful to provide key support for the results presented and to make conclusions more robust and sound. Frequently, however, in other instances, the reviewers seek from authors something that instead is not strictly linked to the work under revision, but focus on the next step of the research and possibly matter of a follow-up paper. While making trouble to the authors, this kind of reviewer’s feedback may create problem also to the handling editors of the manuscript, who often prefer to refer to other reviewers in search of a majority opinion, instead of looking carefully at the recommendations and requests of additional work/data by the original reviewers, and take their own decision.
Updates in Surgery (Giuseppe Remuzzi)
JB: At one point Dr Remuzzi was the only clinical researcher to sit on the editorial board of both The Lancet and NEJM, which is a remarkable political achievement. He has been rotated off NEJM now, I see.
In our experience, the majority of our interactions with our authors, referees and readers are polite and constructive. However, there are unfortunately instances of communications that are more problematic. As editors, we are committed to behaving with professionalism, civility and respect in all our interactions with members of the science community and general public, and we expect the same standard of behavior to be reciprocated by those who interact with us. Our condemnation of all types of egregious behavior, including aggression, discrimination, bullying and harassment, is unequivocal, and such behavior is not tolerated, as is now codified in our portfolio’s policy. Communicating with respect is at the top of the rules of engagement in the editorial and publishing process, as it should be for all interactions in life.
Nature Cancer (unsigned editorial)
Journal Club
As the subject coverage of the 50-year-old list had never been evaluated, the CCJ [Core Clinical Journals] committee began its innovative step-wise approach by analyzing the existing subject scope. To determine whether clinical subjects had changed over the last half-century, the committee collected data on journal usage in hospitals and medical facilities, adding journal usage from Morning Report blogs recording the journal article citations used by physicians and residents in response to clinical questions. Patient-driven high-frequency diagnoses and subjects added contextual data by depicting the clinical environment. The analysis identified a total of 80 subjects and selected 241 journals for the updated Clinical Journals filter, based on actual clinical utility of each journal.
Journal of the Medical Library Association (Michele Klein-Fedyshin and Andrea M. Ketchum)
JB: I’ve used the core clinical journals filter in PubMed at various points over the years. This transition to creating a list of 241 journals that have ‘clinical utility’ is an interesting approach. There are some odd omissions, though. The Lancet is included, but none of the Lancet specialty journals make the grade; the JAMA specialty journals are included, however. I haven’t read the methodology in detail, but the list looks better than the one produced by Nature Index.
One hundred and eighty-six respondents completed our international survey on peer review training. Among respondents, the vast majority indicated they have never received formal training in peer review. A lack of training could therefore explain the less-than-optimal reporting quality of biomedical research and the inability of reviewers to detect major errors and deficiencies in publications.
PLOS ONE (Jessie V. Willis et al)
We acknowledge that the empiric benefits of peer review remain debatable, and it may not filter high-quality from low-quality research as well as we like to believe. However, we believe it is better than an alternative system in which there is no formal process for peer input and constructive critique. While current peer review is mostly hidden, taking place before publication, newer models are needed that include engagement on preprint platforms through comments and discussion, as well as through community peer review forums like PubPeer.
Journal of General Internal Medicine (Christopher M. Booth, Joseph S. Ross, and Allan S. Detsky)
It is crucial that actors in different domains take responsibility for improvements and work together to ensure that high-quality outputs are incentivized and rewarded. If one is fixed without the other (e.g., researchers focus on high-quality outputs [individual level] but are incentivised to focus on novelty [structural level]), then the problems will prevail, and meaningful reform will fail. In outlining multiple positive changes already implemented and embedded, we hope to provide our scientific community with hope, and a structure, to make further advances in the crises and revolutions to come.
Communications Psychology (Max Korbmacher et al)
Here, I conduct a detailed analysis of three Monsanto review articles and a five-article journal supplement for which detailed information from company emails is publicly available following litigation over Roundup. All the articles had external, but not Monsanto authors, and ghostly practices including ghost authorship, corporate ghost authorship and ghost management were evident in their development. There was clear evidence of ghostwriting – that is, drafting of the manuscript by non-authors – in only two cases. I found no evidence of undeserving authorship among the external authors. The articles complied with the disclosure requirements of their journals, save for the journal supplement.
Accountability in Research (Alastair Matheson)
And finally...
This news story made me sit up and take notice, for obvious reasons: Academics turn to paid newsletters for scholarly connection.
Hoel said in his six months of working on his Substack full-time, it has replaced 80 percent of his Tufts salary. He acknowledged the move would sound “a bit extreme” to the average person. While not every professor leaves their job for a newsletter, a growing number of academics are turning toward the service, according to Substack. From July 1, 2022, to July 1, 2023, the site saw a 107 percent jump in academic publishing, representing “thousands” of new publications, the company said. There also was a 42 percent increase in academic paid subscriptions. The increase of academics on the site is at a “tipping point,” said Clyde Rathbone, partnerships manager at Substack.
I’ve considered making this newsletter a paid product, but my gut instinct tells me that’s not the right approach. Earlier this week I listened to a podcast interview with Nathan Barry, the founder of ConvertKit, which is the tool I use to send this newsletter. Nathan said that he’s guided by the following principles:
- Show up every day
- Relentlessly pursue goals
- Learn constantly
- Share learnings with the world
- Default to generosity
Those sound like excellent principles to me.
Until next time,
James
P.S. If you agree with “share learnings with the world” and “default to generosity”, perhaps you would be kind enough to leave a testimonial or forward this email to your colleagues.