Title: LimiX-2M: Addressing Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

Abstract

While tabular foundation models (TFMs) are increasingly competitive with tree ensembles, they frequently suffer from inefficiencies in computational resource usage. Standard approaches that utilize affine scalar tokenization constrain each feature to a one-dimensional channel for value variation. Consequently, feature identifiers and positional signals fail to enhance the degrees of freedom within feature values, leading to redundant hidden states and limited sensitivity to values in the initial layers. To address these issues, we introduce a unified tokenize-and-route framework designed to strengthen TFMs. Our approach features RaBEL, which transforms each scalar into compact, localized Radial Basis Function (RBF) features—optionally gated by exponentials—to enhance conditioning and boost the effective rank in shallow layers. Additionally, we implement a reordered bidirectional block, S$\rightarrow$N$\rightarrow$F, which synchronizes computational processes with the readout mechanism. This is achieved by aggregating cross-sample context prior to feature mixing and employing attention pooling. The resulting model, LimiX-2M, comprises only 2 million parameters yet surpasses larger baselines such as TabPFN-v2 and TabICL across standard tabular benchmarks, all while lowering both training and inference expenses. These findings underscore that value-aware tokenization and readout-aligned routing are critical factors in optimizing the balance between accuracy and efficiency in TFMs. Model checkpoints and inference code can be accessed at https://github.com/limix-ldm-ai/LimiX.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC