Title: The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

Abstract:

While the digitalization of healthcare has produced an abundance of clinical text suitable for powering AI-driven applications, German biomedical language models are currently hindered by outdated architectures and insufficient training data. To address this gap, we introduce ChristBERT (Clinical- and Healthcare-Related Issues and Subjects Tuned BERT), a suite of domain-adapted German language models based on the RoBERTa architecture. These models were developed using a comprehensive 13.5GB dataset comprising scientific publications, clinical documentation, health-focused web content, and translated clinical resources.

This study explores the efficacy of various domain adaptation techniques within the context of German clinical natural language processing (NLP). We conduct a comparative analysis of three approaches: training from scratch, continued pre-training, and domain-specific vocabulary adaptation. The performance of the resulting models was assessed across five benchmarks, including three medical named entity recognition tasks and two text classification tasks.

ChristBERT demonstrates superior performance, surpassing both existing general-purpose and medical-specific German language models on four out of the five evaluated benchmarks, thereby setting a new state-of-the-art for German clinical language modeling. Our findings indicate that the most effective adaptation strategy is contingent upon the specific task at hand. Specifically, training from scratch yields the best results for highly specialized clinical texts, while continued pre-training proves more effective for commonly used medical documentation. To facilitate further advancements and applications in German medical NLP, all models have been made publicly available.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC