Title: GENEB: The Challenges of Comparing Genomic Models

Abstract

Assessing advancements in genomic foundation models remains a significant challenge, primarily due to fragmented benchmarking systems, inconsistent evaluation protocols, and the reliance on task-specific reporting. Consequently, assertions regarding a model’s superiority or general applicability are frequently difficult to compare directly. To address this, we present GENEB, a comprehensive diagnostic benchmark designed to evaluate frozen representations from 40 genomic foundation models. This framework assesses performance across 100 distinct tasks organized into 13 functional categories, utilizing a unified probing-based protocol that encompasses few-shot scenarios.

GENEB facilitates controlled comparisons by isolating variables such as model scale, architecture, tokenization methods, and pretraining data, while also highlighting trade-offs at the task level. Our findings reveal that aggregate leaderboards are inherently unstable; model rankings fluctuate dramatically depending on the task category. Furthermore, the analysis indicates that increasing model scale yields only modest and inconsistent improvements, whereas alignment between architecture and pretraining data often proves more critical than the sheer number of parameters. These insights underscore the shortcomings of existing evaluation methodologies and establish GENEB as a standardized reference for principled comparison and category-aware model selection within genomic machine learning.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC