Title: WildCat: Near-Linear Attention in Theory and Practice

Abstract:

We present WildCat, a novel method for compressing neural network attention mechanisms that delivers high precision at a significantly reduced computational cost. Although attention is fundamental to contemporary network designs, its deployment is often hindered by substantial resource demands, as complexity typically increases quadratically with the input sequence length $n$. WildCat circumvents these quadratic bottlenecks by focusing attention on a small, weighted coreset. We identify this coreset via a rapid yet spectrally precise subsampling technique known as randomly pivoted Cholesky, assigning optimal weights to each element to minimize reconstruction error. Notably, for bounded inputs, WildCat achieves an approximation of exact attention with a super-polynomial error decay rate of $O(n^{-\sqrt{\log(\log(n))}})$, all while operating in near-linear time, $O(n^{1+o(1)})$. Previous practical approximations generally fail to offer such error guarantees or demand quadratic runtime to maintain comparable fidelity. To validate these theoretical advantages, we provide a GPU-optimized PyTorch implementation and conduct extensive benchmark experiments. These tests highlight WildCat’s efficacy across several domains, including image classification, image generation, and the compression of the KV cache in language models.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC