Title: KletterMix: Ascending to High-Quality German Pretraining Data

Abstract

While high-quality pretraining data is a critical component of modern language models, German-language resources lag significantly behind their English equivalents. These German resources are typically characterized by smaller sizes, less rigorous curation, sparse documentation, and a lack of validation through controlled training experiments. To address this gap, we present KletterMix, a high-quality German corpus designed for language model pretraining and annealing. Intended as a reusable artifact for the NLP and modeling communities, KletterMix is constructed by translating a state-of-the-art English pretraining corpus into German, maintaining document boundaries, metadata, source structure, and topical diversity. This approach generates a German corpus that matches the scale and variety of contemporary pretraining datasets, while also facilitating direct comparison with its English source. We provide comprehensive documentation of the dataset through extensive corpus-level analyses, covering translation quality, document length distributions, topic coverage, source composition, and geographic metadata. Our evaluation using COMETKiwi indicates that the translated documents maintain strong quality across various domains, implying that meticulous translation can retain much of the semantic and stylistic depth of the original text. Furthermore, we assess KletterMix’s utility as training data. Controlled pretraining and annealing ablations comparing KletterMix to established German corpora reveal that models trained on this dataset achieve measurable gains in German-language downstream evaluations. These findings highlight that carefully curated translated data can significantly enhance the German pretraining data ecosystem.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC