Many of us have been using LLMs for our research - whether brainstorming, writing, analysis, and so much more. For me, it’s been incredibly productive.
Yet as a tenure track faculty member, it’s difficult to believe that our journals will withstand the strain.
That’s why a group of us came together to discuss this impending crisis and a potential solution - to fight fire with fire and infuse AI into peer review.
Our proposal, which deploys LLMs as a first-line reviewer, allows authors to respond to this journal managed review with a response note. The subsequent reviewers receive the original manuscript, the AI review, and the response note. Our analytical model shows that this can decrease turnaround times and improve decision accuracy.
As reviewers are already burdened, and some are using AI to generate reviews - what we call “Shadow AI” - we are in a regime where authors may not be receiving the high-quality feedback they hope for, and decisions may feel arbitrary. A process governed and managed by the journal, with a high-quality AI review on criteria determined by the journal, allows the human reviewers to contribute in a meaningful way.
We wrote this as a call to action, to experiment with AI reviews in the information systems discipline. There are plenty of open questions including: should the AI reviewer be made publicly accessible, what are the behavioral effects on reviewers, what might we expect with respect to gaming, and others. We discuss some of these questions and invite others to study these topics, as we will certainly do.
This type of work is never possible without an exceptional team. Our captain Hemant Bhargava, and first mate Zhe Zhang, model extraordinaires Daniel Z and Pantelis Loupos, valiant experiment team Laura Brandimarte and Frank Li, and AI enabled reviewer in chief VC.
We are eager to run experiments to see how this works. We will be launching an experiment as part of WORTH 2027, with the best-in-class reviews from refine.ink.
Initial submissions are due September 14, 2026 - please submit your best work and reach out to me if you’re interested in being a reviewer.