Title: Bounded Hyperbolic Tangh: A Robust and High-Performance Substitute for Pre-Layer Normalization in Large Language Models

Abstract:

Pre-Layer Normalization (Pre-LN) has emerged as the standard normalization technique for large language models (LLMs), playing a pivotal role in ensuring stable pretraining and facilitating effective transfer learning. Despite its widespread adoption, Pre-LN suffers from significant computational overhead due to repeated statistical calculations. Furthermore, it is susceptible to the "curse of depth," a phenomenon where the magnitude and variance of hidden states escalate with additional layers, leading to training instability. While normalization-free approaches like Dynamic Tanh (DyT) offer improved throughput, they often lack robustness in deeper architectures.

To simultaneously resolve issues of stability and efficiency, this study introduces Bounded Hyperbolic Tanh (BHyT), a seamless alternative to Pre-LN. BHyT integrates a tanh activation function with explicit, data-driven input constraints to maintain activations within a non-saturating range. This mechanism effectively curbs the depth-wise expansion of activation magnitude and variance, backed by a theoretical guarantee of stability. In terms of efficiency, BHyT calculates exact statistics only once per block and substitutes a secondary normalization step with a computationally lightweight variance approximation.

Experimental results indicate that BHyT enhances both stability and efficiency during the pretraining phase. Compared to RMSNorm, BHyT accelerates training by an average of 1.6% and increases token generation throughput by an average of 1.77%. Additionally, it preserves strong performance in both pretraining-only and post-Supervised Fine-Tuning (SFT) evaluations across various benchmarks for language understanding and reasoning.

Code is available at: https://github.com/MLAI-Yonsei/BHyT

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC