FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization
Title: FrontierOR: Assessing LLM Capabilities in Crafting Efficient Algorithms for Large-Scale Optimization
Abstract:
While large language models (LLMs) are gaining traction in generating solver code and building optimization models, real-world operations research presents a more demanding challenge: the creation of scalable algorithms that leverage specific problem structures to surpass the performance of standard "formulate-and-solve" approaches. Current benchmarks fall short in this regard, typically relying on simplified or small-scale examples that do not reflect the complexity of actual industrial problems. To address this gap, we present FrontierOR, one of the initial benchmarks dedicated to systematically assessing LLMs' ability to design efficient algorithms for realistic, large-scale optimization scenarios.
FrontierOR comprises 180 tasks sourced from methodologically varied studies published in premier operations research journals. Each task features standardized problem instances alongside a hidden evaluation suite verified by experts. We tested seven distinct LLMsāranging from cutting-edge and cost-efficient models to open-source variantsāunder both one-shot and test-time evolution conditions. Our findings indicate that even state-of-the-art models face significant hurdles in transitioning from executable code to truly efficient optimization strategies. Specifically, the top-performing one-shot model exceeded Gurobiās performance in merely 31% of cases regarding both computational efficiency and solution quality. Furthermore, sophisticated coding agents utilizing test-time evolution managed to succeed in only 50% of the most challenging tasks.
FrontierOR serves as a robust platform for evaluating LLM-driven algorithm design, facilitating the systematic testing of whether future models can progress beyond mere formulation correctness to deliver feasible, high-quality, and efficient algorithms. The associated code and data are available at https://github.com/Minw913/FrontierOR.
Source: arXiv Generated at: 2026-06-02 00:00:00 UTC
