Title: A Distributional Perspective on the Pre- and Post-Temperature Dynamics of Creative LLM Output

Abstract

Current methods for evaluating the creativity of large language models (LLMs) without reference data typically depend on metrics such as perplexity, entropy, and the top-1 probability margin. However, this study demonstrates that a significantly more potent predictive signal exists one step earlier in the generation process: specifically, in the manner in which sampling temperature alters the model’s token distribution prior to the selection of the subsequent token.

We analyzed 500 open-ended creative prompts using the Llama-3.1-8B-Instruct model across three temperature settings ($T \in {0.3, 0.8, 1.5}$). By employing a single per-token feature derived from this distributional reshaping, we achieved a Spearman correlation ($\rho$) of 0.918 with creativity rankings judged by an averaged ensemble of GPT-4o and Gemini-2.5-Pro ($n=500$), and 0.870 with a human-majority ranking from three raters ($n=150$).

In contrast, four conventional reference-free baseline metrics—self-perplexity, mean predictive entropy, top-1 margin, and gzip compression ratio—maxed out at approximately $|\rho| \approx 0.76$ for both ground-truth sources. This represents a performance gap of +0.165 against the averaged LLM judges and +0.110 against the human-majority rankings. These improvements are substantial, exceeding the variance observed among the baseline methods themselves.

Furthermore, the two ground-truth evaluation panels showed strong agreement with each other ($\rho=0.83$), a figure that surpasses the inter-human reliability ceiling of $\rho=0.77$. This indicates that the high correlation of our proposed method is not an artifact of judge noise.

Mechanistically, the superior performance stems from a distinct distributional signature associated with the incoherence regime. At a temperature of $T=1.5$, the cumulative-mass width $n_{95}(q)$ expands dramatically from approximately 1 to roughly 131 tokens. Additionally, post-temperature probability mass leaks away from the pre-temperature top-90% plausible set by about 13 percentage points. While the per-token aggregates used in this study do not effectively distinguish between the coherent regimes of $T=0.8$ and $T=0.3$, separating these two settings remains the domain of sequence-level features.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC