Title: Enhancing Manuscript Development Through AI Feedback: An Analysis of 20 Computer Architecture Submissions

Abstract:

As artificial intelligence accelerates scientific progress, it simultaneously drives an exponential increase in the volume of AI-assisted research publications. This surge is placing significant pressure on traditional peer review systems, prompting some to adopt AI-generated critiques—a practice that is both widespread and often subtle. While the broader academic community has yet to reach a consensus on the ethical implications regarding confidentiality, quality assurance, and fairness, the debate persists. Rather than focusing solely on these controversies, this study poses a pragmatic question: Can AI feedback enhance the quality of manuscript drafts?

To answer this, we conducted a case study involving 20 computer architecture papers, representing a spectrum of submission histories. Our goal was to evaluate the alignment between AI and human reviews using a specific set of defined metrics. For this purpose, we developed a web-based tool, AI-Paper-Review, available at https://github.com/unarylab/ai-paper-review. This tool aggregates reviews from a diverse array of AI models, clustering and prioritizing comments based on their frequency and significance. It also facilitates the alignment of AI-generated insights with human feedback to enable quantitative validation.

Our findings indicate that AI critiques address a substantial portion of issues identified by human reviewers, while also highlighting gaps in human analysis. However, this work does not advocate for the use of AI in formal peer review at this time. Instead, it aims to investigate (1) the capacity of AI feedback to refine paper drafts and (2) the capabilities and constraints of AI-driven review processes. By releasing both the tool and the associated case study data, we hope to stimulate further research in this area. Please note that utilizing AI for actual peer review would contravene the ethical guidelines established by major academic conferences and journals.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC