Listen to the episode — Welcome, AI Reviewer Two, on This IS Research →
The day our episode on AI peer review came out, my coauthor and I happened to be working through the comments that we had received from Refine.ink. It was a helpful process. While there were some things the AI tool made mistakes on, other comments made us clarify necessary assumptions. On the whole, they improved the paper.
There are a lot of theories of reviewing and editing – see, for example, my colleague Edgar Whitley’s Information Systems Research editorial on this topic. Editors and reviewers can be coaches, coordinators, facilitators, helpers, tutors, advisors, developmental gatekeepers, stewards, and much more. Whatever the role, you often hope for something constructive after you submit a paper. And the last three papers I’ve submitted, I’ve received that.
But I feel like I’m one of the lucky ones. More and more of my friends and colleagues have had pretty disheartening experiences, with reviewers who do not seem to care about truly understanding the paper, responding with comments or criticism that suggests they are not engaging with the paper.
It’s really easy to be bogged down by bad feedback. And when you receive such feedback after months waiting, especially with the pace of other parts of life, it can be quite disappointing.
It is difficult to speed up the review process. And we need to figure out a way to decrease the burden on human reviewers, while using the new AI tools at our disposal to improve our research. That’s why, on This IS Research, we’ve described our proposal to fight fire with fire and infuse AI tools into the review process.
We’ve got a model in the paper, and it is dependent on multiple parameters, which we will test very shortly with your support. Every author submitting to the WORTH 2027 conference will get a Refine.ink review of their submission. You can respond to the review, and we will randomize whether the review and your response will be sent to zero, one, or two of the paper’s reviewers. For folks who want to serve as reviewers, you can also sign up to be a part of this experiment.
Thanks to Jan Recker and Nick Berente for the opportunity to discuss our work, and looking forward to better understanding the role of AI review. If you’re interested in reviewing, please sign up. If you’re an author, send us your best work – submissions are due October 12, 2026.