Title: Evaluating Large Language Models on Deterministic Programming Tasks: A Focus on Accuracy, Stability, and Consistency

Abstract

Standard metrics for large language model (LLM) evaluation often rely on single-run accuracy or the probability of eventual success through repeated sampling. However, many real-world deployment scenarios demand stability—defined as the ability to produce consistent outputs across multiple invocations of the same prompt. This study explores the relationship between accuracy and stability in deterministic text-conditioned generation, using programming tasks as a primary testbed. We introduce a rigorous evaluation protocol that measures run-level accuracy, retry-free coverage, and variability per problem.

Our analysis involved 16 models from five distinct provider families, tested against a recent benchmark comprising 100 LeetCode-style problems. Each problem was evaluated using two different prompt templates with five repeated runs, resulting in a total of 16,000 evaluation instances.

The findings reveal that relying solely on run-level pass rates can significantly overstate true retry-free coverage, with discrepancies reaching up to 17.8 percentage points. This gap is most pronounced in mid-performing systems. While run-level pass rate and perfect stability rate show a strong positive correlation (r=0.985), the pass rate consistently remains higher than the retry-free coverage. In some cases, this disparity is substantial enough to reverse the ranking of closely matched models. Additionally, the impact of prompt variations was found to be model-specific rather than universally advantageous. These results underscore the necessity of incorporating repeated-run stability analysis alongside traditional accuracy metrics to provide a more complete picture of model performance in deterministic generation tasks.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC