Title: Deriving Prospect Theory Phenomena from Bellman Optimality in MDPs Featuring Catastrophic Absorbing States

Abstract:

This study investigates risk-neutral control within Markov decision processes that include an absorbing catastrophic state. We demonstrate that standard Bellman optimality generates three distinct signatures resembling prospect theory, despite the absence of utility curvature, probability weighting, or framing effects, and even though rewards remain linear. These signatures include an S-shaped value function—characterized by convexity near the catastrophe and concavity in distant regions—an endogenous loss-sensitivity coefficient $\lambda^*(S)$ exceeding 1, and a policy reversal indicative of the reflection effect.

Analysis of 495 configurations reveals that the optimal policy exhibits risk-averse behavior near the catastrophic state in positive-drift (growth) scenarios, even when the risky action offers a higher immediate expected value. Conversely, in negative-drift (decline) scenarios, the agent becomes risk-seeking near the catastrophe, despite the safe action resulting in a lower immediate expected loss. We establish a closed-form formula for the asymptotic loss-aversion plateau, $\bar{\lambda}$, which is determined solely by the win probability $p$, the discount factor $\beta$, and payoff asymmetry $r = |\Delta_\ell/\Delta_w|$. This analytical solution aligns with numerical results with an $R^2$ value of 0.999.

The mechanism driving these phenomena does not strictly require asymmetric payoffs. In a comprehensive sweep across $(p,\beta)$ values at three asymmetry levels, the contribution of asymmetry to $\bar{\lambda}$ exceeding unity was minimal, with a median share of 4.6% at $r = 1.25$ and rising to 13.9% at $r = 2$. In every tested case, the contribution from the boundary conditions surpassed that of asymmetry.

These findings hold robust under various computational and environmental conditions. Tabular Q-learning, a model-free approach, reproduces the optimal value function $V^*$ with correlations of 0.98 in growth regimes and 1.00 in decline regimes. Furthermore, the results persist under stochastic transitions involving Gaussian, heavy-tailed Student-$t_3$, and asymmetric skew-normal noise, up to 50% of the step size. In these noisy environments, the asymptotic plateau tracks the closed-form prediction within 0.41% for safe-channel noise and within 9.6% for risky-channel or dual-channel noise. Ultimately, these results identify absorbing failure states as a sufficient structural mechanism for generating prospect-theory-like behavior under optimal control.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC