Title: GottBERT: A Dedicated German Language Model

Abstract:

The landscape of natural language processing (NLP) has been transformed by pre-trained language models, particularly following the emergence of BERT and its enhanced counterpart, RoBERTa. Although early studies concentrated primarily on English, there is a growing recognition that monolingual models often outperform multilingual alternatives regarding pre-training efficiency, resource optimization, and performance on specific downstream tasks. Even as prompt-based large language models gain traction, BERT-style architectures continue to hold significant value due to their computational efficiency.

In this study, we introduce GottBERT, the first RoBERTa model designed exclusively for the German language. It was pre-trained solely on the German segment of the OSCAR dataset. We also examined how filtering the OSCAR corpus influences model outcomes. Utilizing the fairseq framework and standard hyperparameters, we trained GottBERT and assessed its capabilities across five distinct tasks: two Named Entity Recognition (NER) benchmarks (Conll 2003 and GermEval 2014) and three text classification datasets (GermEval 2018 fine and coarse, and 10kGNAD). We benchmarked GottBERT against existing German-specific BERT models as well as two multilingual models, evaluating performance via accuracy and the $F_{1}$ score.

Our results indicate that both the base and large versions of GottBERT are highly competitive. Notably, GottBERT achieved the highest scores among base models in four out of the six evaluated tasks. Unexpectedly, our analysis revealed that the applied data filtering had negligible impact on the final results. To foster progress within the German NLP research community, we are making the GottBERT models available under the MIT license.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC