Title: HalleluBERT: Empowering Meaningful Tokens with Their Proper Weight

Original: arXiv:2510.21372v2 Announce Type: replace Abstract: Transformer-based models have advanced NLP, yet Hebrew still lacks a RoBERTa encoder that is trained at scale and released in both base and large variants. We present HalleluBERT, a RoBERTa-based encoder family trained from scratch on 49.1~GB of deduplicated Hebrew web text and Wikipedia using a Hebrew-specific byte-level BPE vocabulary. On native Hebrew benchmarks for named entity recognition (BMC, NEMO) and sentiment classification (SMCD), HalleluBERT outperforms monolingual and multilingual baselines, and yields the highest unweighted mean score across the three benchmarks. We release model weights and tokenizer under the MIT license to support reproducible Hebrew NLP research.

Rewrite: While Transformer architectures have significantly propelled the field of Natural Language Processing, the Hebrew language remains underserved by the absence of a large-scale, publicly available RoBERTa encoder offering both base and large configurations. To address this gap, we introduce HalleluBERT, a new family of RoBERTa-based encoders developed entirely from the ground up. This model was trained on a curated dataset comprising 49.1 GB of deduplicated Hebrew content sourced from Wikipedia and the web, utilizing a specialized byte-level Byte Pair Encoding (BPE) vocabulary tailored specifically for Hebrew.

In evaluations on standard Hebrew benchmarks for sentiment analysis (SMCD) and named entity recognition (BMC and NEMO), HalleluBERT surpasses both monolingual and multilingual baseline models, achieving the highest unweighted average performance across all three tasks. To foster transparent and reproducible research in Hebrew NLP, we have made both the tokenizer and model weights available under the permissive MIT license.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC