Title: EURO-5K: Evaluating the Impact of Domain Pretraining in Transformer Models for EU Reporting Obligation Extraction

Abstract

Accurately identifying and minimizing the regulatory reporting burden requires the precise extraction of reporting obligations from European Union legislation. This task is particularly difficult because reporting requirements often resemble other structural provisions, demanding a nuanced legal comprehension that general natural language processing (NLP) techniques often lack. Currently, the field suffers from a shortage of specialized datasets featuring clear guidelines, as well as a lack of comparative analysis regarding extraction paradigms and domain adaptation strategies.

To address these gaps, we introduce EURO-5K, a comprehensive corpus comprising sentence-level reporting obligations alongside challenging negative examples drawn from 136 EU legislative acts. Using this dataset, we conduct a rigorous comparison between discriminative token-classification models (such as BERT-style architectures) and generative span-extraction models (LLMs). Our evaluation encompasses full fine-tuning and parameter-efficient QLoRA, benchmarking these against baseline methods including pattern matching, dependency-based extraction, and few-shot prompting.

Our findings indicate that fully fine-tuned generic and legal-specific BERT models yield comparable performance, both achieving an F1 score of 0.89. Similarly, fine-tuned LLMs demonstrate accuracy levels on par with encoder-based models for sentence-level extraction. While legal pretraining provides only marginal improvements for generative models, it proves significantly advantageous when adaptation capacity is limited; specifically, parameter-efficient tuning of Legal-BERT surpasses its generic counterpart. Furthermore, learning curve analyses reveal that legal pretraining facilitates faster early-stage learning even with scarce data. All tested approaches reach convergence at approximately 3,000 samples, after which additional data yields diminishing returns, thereby confirming the sufficiency of the dataset.

Cross-dataset assessments on two external regulatory corpora confirm that our models function as specialized extractors for reporting obligations rather than general regulatory classifiers. In conclusion, we release the EURO-5K dataset, along with the trained models and an interactive demonstration featuring explainability visualizations and structured RDF export capabilities. These resources highlight how both model paradigms and parameter-efficient training methods offer practical solutions for automating regulatory compliance.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC