Title: Markov Chain Decoders Surmount the Heavy-Tail Constraints of Lipschitz Generative Models

Abstract:

Heavy-tailed distributions are ubiquitous in fields such as risk modeling, network traffic analysis, and performance evaluation. This characteristic presents a significant obstacle for contemporary deep generative models. Conventional Variational Autoencoders (VAEs) rely on Gaussian decoder likelihoods paired with Lipschitz-constrained neural networks. This architectural combination is inherently unable to generate heavy-tailed outputs: the Gaussian tail exhibits exponential decay, while Lipschitz continuity restricts the decoder from sufficiently amplifying rare events originating in the latent space to counteract this decay.

In this work, we offer a theoretical analysis of this limitation and validate it through controlled empirical tests using synthetic Pareto data. These tests span a range of tail indices ($\alpha$ $\in$ {2, 3, 5, 30}) and dimensions ($d$ $\in$ {1, 5, 10}). To address this issue, we substitute the standard Gaussian decoder with a Phase-Type (PH) distribution derived from Markov chains, while maintaining the original encoder, latent space, and training protocol unchanged. Because PH distributions can approximate any positive-valued distribution with arbitrary precision—including heavy-tailed families—they offer a robust alternative.

Our experiments indicate that the PH-based model significantly outperforms the Gaussian baseline for heavy-tailed data, reducing tail Kolmogorov-Smirnov distance by a factor of up to 6 and extreme quantile error by up to 10. These findings confirm that incorporating Markov chain-based distributions into the decoder of a generative model provides a principled and practically effective resolution to the challenge of heavy-tail generation.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC