Citation Grounding: Detecting and Reducing LLM Citation Hallucinations via Legal Citation Graphs

Large language models (LLMs) frequently generate inaccurate legal citations, a phenomenon that includes inventing statute references, pointing to repealed laws, and mixing up legal jurisdictions. Despite the prevalence of this issue, there has been no automated, scalable method to quantify or mitigate these hallucinations. To address this gap, we introduce Citation Grounding (CG), a novel metric designed to validate LLM-generated citations against a comprehensive ground-truth citation graph. This graph was constructed from an analysis of 100.8 million Ukrainian court decisions, encompassing 502 million edges and 21,736 unique statute nodes.

CG operates through three distinct components to provide a differential diagnosis of hallucination types: citation precision, which verifies the existence of the cited provision; citation relevance, which assesses contextual appropriateness; and citation temporality, which confirms whether the citation was valid at the specific date in question.

In an empirical evaluation involving 100 Ukrainian legal queries, we tested five different systems: four commercial LLMs accessed via AWS Bedrock (Claude Haiku 4.5, Mistral Pixtral Large, Amazon Nova Pro, and Amazon Nova Lite) and one production system augmented with Retrieval-Augmented Generation (RAG). The results showed CG scores ranging from 0.791 to 0.873, with between 13% and 21% of the generated citations identified as hallucinated.

To mitigate these errors without relying on human annotation, we propose Citation Grounding DPO (CG-DPO). This approach algorithmically creates preference pairs by intentionally corrupting verified citations found in real court decisions using four specific strategies. When applied to a dataset of 2,244 court decisions, a Qwen2.5-7B-Instruct model fine-tuned with LoRA achieved a mean validation accuracy of 98.5% in distinguishing between correct and corrupted citations. This performance was supported by a reward margin of +14.9 and a standard deviation of less than 0.3 percentage points across three random seeds. The authors have made the citation graph, the evaluation framework, and the CG-DPO dataset available as open resources.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC