Title: Moving Past Simple String Matching: A Semantic Approach to Evaluating PDF Table Extraction

Abstract:

The accurate extraction of tabular data from PDFs is a cornerstone of large-scale scientific data mining and the development of robust knowledge bases. However, current evaluation methodologies predominantly depend on rule-based metrics, which often overlook the semantic equivalence of the extracted content. To address this gap, we introduce a benchmarking framework that utilizes synthetically generated PDFs. These documents feature precise LaTeX ground truth and incorporate tables from arXiv to guarantee both realistic complexity and diversity.

Our primary methodological innovation involves the deployment of an "LLM-as-a-judge" system for semantic table evaluation. This approach is embedded within a matching pipeline designed to handle inconsistencies inherent in parser outputs. We validated our method through a human study involving over 1,500 quality assessments of extracted table pairs. The findings indicate that LLM-based evaluation correlates significantly more strongly with human judgment (Pearson r=0.93) than traditional metrics such as Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70).

Furthermore, our assessment of 21 modern PDF parsers across 100 synthetic documents, which contained a total of 451 tables, highlighted substantial variations in performance. These results provide actionable insights for choosing the most suitable parsers for tabular data extraction and establish a reproducible, scalable evaluation standard for this essential task.

Code and Data: https://github.com/phorn1/pdf-parse-bench Metric Study and Human Evaluation: https://github.com/phorn1/table-metric-study

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC