Title: PortBERT: Charting the Course for Efficient Portuguese Language Modeling

Transformer architectures have established themselves as the backbone of contemporary natural language processing; however, there is a notable scarcity of language-specific models that prioritize efficiency alongside performance. In the domain of Portuguese, existing efforts have predominantly concentrated on maximizing scale or accuracy, frequently overlooking the critical aspects of training and deployment efficiency. To address this gap, this study presents PortBERT, a collection of RoBERTa-derived language models tailored for Portuguese that seek to strike an optimal balance between computational efficiency and model performance.

The models were trained from the ground up utilizing fairseq, employing stable pre-training protocols compatible with both TPU and GPU hardware. The training corpus comprised more than 450 GB of data, specifically deduplicated and filtered subsets of mC4 and OSCAR23 sourced from CulturaX, and utilized byte-level BPE tokenization. We introduce two distinct versions: PortBERT base and PortBERT large.

To assess their capabilities, we evaluated these models using ExtraGLUE, a benchmark suite consisting of translated tasks from GLUE and SuperGLUE. The results indicate that PortBERT variants are highly competitive, performing on par with or better than current monolingual and multilingual alternatives. Furthermore, beyond standard accuracy metrics, this work provides a detailed analysis of training durations, inference latency, and fine-tuning throughput, offering valuable practical perspectives on efficiency. By focusing on the often-overlooked tradeoffs between compute resources and performance in Portuguese NLP, PortBERT adds a significant dimension to existing research. All model checkpoints are publicly available on Huggingface, alongside fairseq versions, to facilitate continued research and real-world application.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC