Title: HalfNet: Randomized Neural Networks with Learned Subspace Geometry

Abstract:

While prior research has frequently explored neural networks featuring weights fixed to random values sampled from specific distributions, such as $N(0, I)$, our study introduces HalfNet. This approach samples random weights from $N(0, \Sigma)$, where the covariance matrix $\Sigma$—which dictates the distribution's geometry—is derived via a low-rank factorization learned directly from data.

Evaluations on the MNIST and CIFAR-10 datasets reveal that HalfNet achieves performance parity with fully trained multilayer perceptrons, despite utilizing a significantly reduced parameter count. Spectral analysis further suggests that the core predictive capability of neural networks stems from the geometric structure of their weight space rather than the exact values of individual parameters. We also note that accuracy correlates smoothly with rank.

It is important to clarify that HalfNet does not merely serve as an architectural modification to impose low-rank constraints. Instead, it functions as a data-dependent random embedding, a mechanism that can alternatively be understood through the lenses of supervised metric learning, random features, or kernel methods.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC