Title: EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

Abstract:

While Large Language Models (LLMs) have significantly enhanced the capabilities of Text-to-SQL—facilitating natural language interaction with databases—current evaluation frameworks remain limited in scope. Prominent benchmarks like Spider, BIRD, and Spider~2.0 primarily assess schema generalization, performance on large-scale databases, and realistic workflow adherence. However, these benchmarks largely neglect enterprise-specific contexts, where generating accurate SQL queries often hinges on accessing private business intelligence, including internal metrics, established reporting standards, and organizational protocols.

To address this gap, we present EntSQL, a benchmark specifically designed for enterprise-oriented Text-to-SQL tasks that evaluates a model's ability to ground long-context information within proprietary business documents. The dataset comprises 1,066 aligned semantic examples in both Chinese and English, spanning five distinct business domains. The majority of these examples demand domain-specific knowledge that extends beyond the immediate question and database schema, often necessitating the construction of complex SQL structures.

Our evaluation results underscore the significant challenges involved in this domain. When provided with lengthy, long-form documents, the top-performing system achieved an execution accuracy of merely 15.9% on English inputs, illustrating the substantial difficulty models face when attempting to ground SQL generation in enterprise-specific knowledge.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC