Journalology #134: AI and peer review
Hello fellow journalologists,
I’ve been keeping an eye on the news wires over the past few weeks, which were thankfully rather quiet over the Christmas break. Today’s newsletter includes a short list of articles to help you to quickly get up to speed as you return back to work.
If you find the newsletter to be helpful, please hit the ‘like’ button at the top of the email; this will help teach the Substack algorithm to promote this newsletter to new readers. Thank you!
Put pressure on publishers to follow best practice — external regulation is the answer
I propose that academic journals submit to independent regulation through an international quality-management standard, known as ISO 9001. Organizations certified as ISO-9001-compliant must demonstrate operations that are customer-focused, committed to continual improvement and underpinned by systematic management approaches and evidence-based decision-making.
ISO 9001 is a widely adopted standard — one million companies and organizations in more than 180 countries are certified, including suppliers of cell lines and other reagents. It makes sense for journals to meet the same standard, sending a clear signal that they embrace the same regulations as many authors and readers do.
JB: If you read one article in this week’s newsletter, read this opinion piece by Jennifer Byrne, which was published in Nature a few days ago. The argument is compelling and would help to solve some of the challenges facing the publishing community as we head into 2026.
At the moment Web of Science acts as a regulator of sorts. If it decides to delist a journal and to remove its impact factor, then submissions will likely fall. However, the process is opaque, with users left to speculate why a journal has been “editorially delisted”. It’s also binary — journals are either in or out — which is a blunt tool that lacks sophistication.
Journal certification that follows internationally recognised standards would help authors and funders to identify high quality journals independent of Journal Impact Factor or CiteScore.
Ideally, there would be some kind of grading system (a bit like a personal credit score), rather than a binary system, that focuses on editorial process rather than on business model.
Do you support Jennifer Byrne’s proposal? Please leave a comment by clicking the button below.
AI can significantly enhance the quality of research by identifying statistical errors and assisting in the identification of papers that may require retraction or correction. Tools are available that can quickly find inconsistencies in data, making research more accurate and reliable while saving time for reviewers and editors. Researchers, editors, and publishers are encouraged to start using these tools responsibly and create clear standards for their use. Training and collaboration are needed to ensure AI is applied correctly and ethically. Still, AI can detect mistakes, it cannot fully understand context or make final ethical decisions. For an optimal balance, AI can serve as a supportive tool while humans perform the final evaluation. This collaboration enhances RI and strengthens trust in scientific publishing.
JB: This review article may be helpful for editors and publishers who want to understand how AI could be used to enhance peer review. This will be one of the hot topics of 2026. The big challenge will be to independently validate, in an open and transparent way, whether new AI peer review tools are effective and can be trusted.
A Cross‐Disciplinary Analysis of AI Policies in Academic Peer Review
The effectiveness of current AI policies for peer review faces challenges, largely because stakeholders struggle to comply effectively or use AI tools as required. Heavy peer review workloads drive editors and reviewers to use AI for efficiency, making fully AI-prohibitive policies highly likely to be violated. Though ethically sound, these policies no longer meet the practical needs in the AI era. Policies allowing limited AI use, however, suffer from operational gaps. Existing guidelines only vaguely state what is banned or allowed but fail to provide editors and reviewers with concrete steps, such as which AI tools are exempt, how to standardise usage, define confidential information, or craft prompts safely. Without these details, such policies urgently need more specific operational guidance.
JB: The discussion section in this research article provides an overview of the practical and ethical challenges of using AI in peer review.
Over the past year, Science has collaborated with DataSeer to evaluate adherence to its policy mandating the sharing of underlying data and code for all published research articles. The initial results are encouraging in that of 2680 Science papers published between 2021 and 2024, 69% shared data. To further improve transparency, a DataSeer reproducibility checklist, which Science tested in a 2025 pilot program, is being integrated into the journal’s protocols. DataSeer’s natural language processing technology scans a paper and generates a prefilled reproducibility checklist. Authors are asked to confirm the entries and make revisions as needed.
JB: You can read the accompanying blog post here, which compares the Science family’s Open Science Metrics with those previously published by PLOS and Taylor & Francis. The AAAS journals come out on top. This could be because: (a) their data sharing policies are implemented in a more robust manner than at the other two publishers; (b) authors are more willing to jump through hoops for career-defining papers, published in Science.
If you would like to purchase a group subscription to Journalology for your department or organisation, please click the button below.
If you can’t accurately retract an article in the LLM environment, you can’t accurately correct an article, or put an Expression of Concern on it. There’s no stable, addressable CMS. A correction notice would just be another set of tokens in the system, competing with the uncorrected version.
JB: This opinion piece by Kent Anderson addresses an important question: how do LLMs handle retracted articles? Content is atomised by LLMs and ingested into their knowledge base. If a paper is subsequently retracted, presumably that atomised content remains. It’s hard to know for sure, since LLMs tend to be a proprietary black box.
We suggest three strategies for preventing AI-generated commentaries from polluting the exchange of ideas in peer-reviewed journals. First, it is critical to raise awareness of the established misuse of AI to generate commentaries including via widespread communication to publishers, editors, and authors. Second, we encourage editors and authors to explore, and perhaps consider requiring documentation of, the publication history of commentary writers. While such an approach is clearly insufficient to identify all AI-generated commentaries, it would likely identify many of the most egregious serial cases of AI-generated commentaries, such as the examples described in this article. Third, we encourage publishers and editors to implement AI detection tools in their editorial workflow, particularly for commentaries that may receive less scrutiny than research articles.
JB: This is another example of how AI is being used by some academics to write “letters to the editor” at scale. See also: AI-assisted Letters to the Editor—Scope of a Growing Ethical and Practical Concern, and CORR’s Approach to Managing It.
Is ‘open science’ delivering benefits? Major study finds proof is sparse
Does the open science movement—the push to make research outputs such as articles, data, and software free to read and reuse—produce the benefits its supporters claim, such as accelerating discovery and promoting science literacy? The answer is a qualified yes, according to one of the most comprehensive, multifaceted studies of the complex and divisive issue.
JB: There’s no doubt in my mind that in an ideal world all research would be open. However, we don’t live in an ideal world; there are limited resources available and open access business models have their own limitations. Therefore, we need to question whether the return on investment for open science practices is worthwhile.
The evolution of interdisciplinarity and internationalization in scientific journals
With the publication metadata from OpenAlex, we examine trends in two groups of journals: disciplinary journals in natural sciences, life sciences, social sciences, and multidisciplinary journals that publish articles in multiple fields. Supporting existing perceptions, we find an almost universal trend towards increasing internationalization of both sets of journals. Nevertheless, we find disparities: medicine journals are less international than journals in other disciplines and do not increase their levels of internationalization, whereas physics journals appear to be segregating between those that are international and those that are not.
Beyond openness: Inclusiveness and usability of Chinese scholarly data in OpenAlex
Results show that OpenAlex indexes only 37% of GCJC journals [A Guide to the Core Journals of China] and 24% of their articles, with substantial disciplinary and temporal variation. Metadata quality is uneven: while basic fields such as title and publication year are complete, bibliographic details, author affiliations, and cited references are frequently missing or inaccurate. DOI coverage is limited, and language information is often incorrect, with most Chinese-language articles labeled as English. These findings highlight significant challenges for achieving full inclusiveness and usability in research evaluation and related activities. We conclude with recommendations for improving data aggregation strategies, DOI registration practices, and metadata standardization to enhance the integration of local scholarly outputs into global open infrastructures.
JB: This topic is increasingly important as article output from China grows.
And finally…
There’s been a focus on AI tools for peer review in this week’s newsletter. However, some editors and researchers are resisting this brave new world. For example, an editorial in Journal of Neuroradiology entitled Bias, accuracy, and trust: no GenAI in peer reviewing made the following suggestion:
A simple way to ensure that the reviewer has read the entire text of a manuscript, would be to ask the author to insert a few words or a short scientific sentence out of context in the manuscript, or even a false statement, and ask the reviewers to find it and mention it (page, line) at the beginning of their evaluations.
I think we can safely put this in the “bad ideas” bucket.
Until next time,
James
P.S. Don’t forget to hit the ‘like’ button (shaped like a heart) at the top of this email if you found the newsletter to be helpful. The Substack promotion algorithm responds to these signals, you see. Thank you for your support.
It sounds like a wonderful idea. I do expect that the process of defining the standard itself would be lengthy and contested — given the multiplicity of stakeholders. Yet, for precisely those reasons, worthwhile! This would not be a quick-win undertaking, and I can imagine that certain aspects that may be wishes for ‘standardization’ may prove elusive, while others may surface based on process/oversight core-wisdom from outside publishing.
The strongest advocacy may come from technical service development. The most reticence may come from areas that endure via hierarchical power/discretion and gatekeeping… A core difficulty is that THIS structure (experience =‘s expertise, power =‘s standards gatekeeping) IS the foundation of academia and peer review. It sounds bad… but is it?? Could an activity like this facilitate a worse situation?
There is a danger that this could generate administrative box-ticking, and little beyond. Education already suffers under this yoke. I confess to a belief (born of experience) that the opacity of IF decision making is a blessing in disguise. It’s impossible to fully ‘game’ metrics you don’t know, and can only guess at…
Sometimes, a separation of powers, and minimizing transparency, is the best way to keep everyone on their toes.
Rather than go down the ISO 9001 route – which I’m not confident would solve the problems identified in this piece – might a better approach be for publishers to be more transparent about the work they undertake?
Specifically, publishers could make it clear what integrity checks they perform (plagiarism, image manipulation etc.) along with publishing the peer review reports and the author responses.
This “pay for service” approach would likely find favour with funders, who will be able to see what services they are paying for in a fair, transparent and verifiable way that reflect the real value publishers provide .
For more information on this "pay for service" approach see this blog post from Bodo Stern (HHMI) and Rachel Bruce (UKRI) at: https://www.coalition-s.org/blog/yes-to-transparent-service-fees-no-to-fees-that-charge-authors-to-exercise-their-rights/