Title: Addressing Softmax Limitations at the Extremes: Extreme Value Corrections for InfoNCE

Abstract:

While InfoNCE serves as the foundational objective for contrastive learning, its reliance on softmax extends beyond mere computational ease; it inherently embeds a specific statistical assumption regarding the selection of the highest-scoring instance. Leveraging extreme value theory, this study demonstrates that this underlying assumption frequently diverges from the normalized embedding paradigms prevalent in contemporary contrastive learning frameworks. To address this discrepancy, we introduce \textsc{WEINCE}, a straightforward adaptation of InfoNCE. This method incorporates an endpoint shortfall correction by blending standard softmax logits with anchor-wise online batch statistics, achieving this integration without introducing any additional trainable parameters. Evaluations across five vision benchmarks reveal that \textsc{WEINCE} delivers consistent performance gains in frozen-feature assessments. These findings underscore that a more accurate statistical handling of hard negatives can significantly enhance contrastive objectives.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC