Title: Crystal: Characterizing Relative Impact of Scholarly Publications

Abstract:

Traditionally, the influence of a referenced paper is evaluated by examining its citation context in isolation within the citing document. Although this method targets the most pertinent text, it hinders the ability to make relative comparisons among all the works a paper references. To address this limitation, we introduce Crystal, a framework that employs large language models (LLMs) to jointly rank all cited papers within a single citing article. To counteract the positional bias inherent in LLMs, our approach ranks each list three times using randomized orders, aggregating the resulting impact labels via majority voting. By leveraging the complete citation context rather than assessing citations independently, this joint strategy more effectively distinguishes highly influential references. On a dataset of human-annotated citations, Crystal surpasses the previous state-of-the-art impact classifier, achieving improvements of +9.5% in accuracy and +8.3% in F1 score. Additionally, Crystal enhances efficiency by requiring fewer LLM calls and outperforms existing baselines when utilizing an open-weight model, thereby facilitating scalable and cost-effective analysis of citation impact. A case study focusing on ACL Test-of-Time award-winning papers reveals that Crystal’s impact assessments correspond closely with long-term scientific recognition. We have made Crystal-Bank available, a dataset comprising 46,800 papers with rankings and impact labels, along with the associated code.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC