Title: Achieving Robust Learning in Group DRO Neurons

Abstract:

This study investigates the challenge of training a single neuron using standard squared loss, specifically addressing scenarios characterized by arbitrary label noise and distributional shifts at the group level across a wide range of covariate distributions. Our primary objective is to determine a "best-fit" neuron, defined by parameters $\mathbf{w}$, that maintains high performance even under the most adversarial reweighting of groups. To this end, we tackle a Group Distributionally Robust Optimization (DRO) problem: assuming we have sample access to $K$ distinct distributions $\mathcal p_{[1]},\dots,\mathcal p_{[K]}$, we aim to approximate $\mathbf{w}_$ such that it minimizes the worst-case objective function across all convex combinations of these group distributions $\boldsymbol{\lambda} \in \Delta_K$. The objective function is defined as $\sum{i \in [K]}\lambda_{[i]}\,\mathbb E_{(\mathbf x,y)\sim\mathcal p_{[i]}}(\sigma(\mathbf w\cdot\mathbf x)-y)^2 - \nu d_f(\boldsymbol\lambda,\frac{1}{K}\mathbf1)$, where $d_f$ represents an $f$-divergence that optionally penalizes deviations from uniform group weights, controlled by the scaling parameter $\nu \geq 0$.

We propose a computationally efficient primal-dual algorithm that produces a vector $\widehat{\mathbf w}$ which achieves constant-factor competitiveness against $\mathbf{w}_*$ under worst-case group weighting. Our theoretical analysis directly addresses the inherent nonconvexity of the loss function, thereby establishing robust learning guarantees even when faced with arbitrary label corruptions and distributional shifts specific to individual groups. Furthermore, empirical evaluations using dual extrapolation updates, inspired by our algorithmic framework, demonstrate promising results on benchmarks for large language model (LLM) pre-training.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC