Title: BioBlue: Systematic Runaway-Optimizer-Like LLM Failure Modes on Biologically and Economically Aligned AI Safety Benchmarks for LLMs with Simplified Observation Format

Abstract:

Discussions regarding "runaway optimization" in AI alignment frequently center on reinforcement learning (RL) agents—systems defined by unbounded utility maximization that over-optimize proxy goals, such as the "paperclip maximizer" scenario or specification gaming, often at the cost of other critical factors. It is commonly presumed that Large Language Model (LLM)-based systems present a lower safety risk because they operate as next-token predictors rather than persistent optimizers. To challenge this assumption, we empirically evaluate LLMs within simplified, long-horizon control environments that demand the maintenance of state or the balancing of objectives over extended periods. These benchmarks include single- and multi-objective homeostasis, the management of unbounded objectives subject to diminishing returns, and the sustainability of renewable resources.

Our results indicate that while LLMs frequently act in accordance with instructions and demonstrate a clear grasp of stated goals for numerous steps, they often succumb to structured context loss. This leads to runaway behaviors, such as disregarding homeostatic targets or reducing multi-objective trade-offs into single-objective maximization, thereby violating concave utility structures. These malfunctions appear reliably following initial phases of competent performance and display distinct patterns—including self-imitative oscillations, unbounded maximization, and a regression to single-objective optimization—even when the context window remains largely unused.

Crucially, the issue is not merely a loss of context resulting in incoherence. While LLMs may appear to handle multiple objectives and remain bounded superficially, their behavior under sustained interaction involving complex objectives is systematically skewed toward that of single-objective, unbounded, and poorly aligned optimizers. We propose a hypothesis involving a token-level pattern reinforcement attractor, suggesting that LLMs increasingly base their actions on the token patterns of their recent action history rather than adhering to original instructions. The specific reasons why these failures are predominantly triggered in multi-objective settings remain an open question.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC