Title: A Latent Variable Framework for Scaling Laws in Large Language Models

Abstract

This study introduces a statistical framework grounded in latent variable modeling to analyze the scaling laws governing large language models (LLMs). The motivation behind this work stems from the swift proliferation of diverse LLM families, each characterized by unique architectures and training methodologies, alongside their assessment across a growing array of benchmarks. Such heterogeneity renders a single, global scaling curve insufficient for accurately representing performance variations across different model families and evaluation metrics.

To overcome this limitation, we propose a latent variable modeling framework where each LLM family is linked to a latent variable that encapsulates the shared underlying characteristics of that group. Consequently, a model’s performance on various benchmarks is determined by its latent skills, which are co-determined by both the family-specific latent variable and the model’s own observable features. We present an estimation procedure for this latent variable model and rigorously establish its statistical properties. Furthermore, we have developed efficient numerical algorithms to facilitate estimation and support a range of downstream tasks. Empirically, we validate our approach using 12 widely recognized benchmarks sourced from the Open LLM Leaderboard (versions 1 and 2).

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC