Title: Scaling Pre-training to One Hundred Billion Data for Vision Language Models

Abstract:

This study offers an empirical examination of the capabilities of pre-training vision-language models using a massive dataset of 100 billion examples. We discovered that on standard benchmarks focused on Western-centric classification and retrieval—such as COCO Captions—model performance tends to plateau at this scale. However, tasks requiring cultural diversity exhibit significantly greater improvements when utilizing this 100-billion-scale web data, largely due to its extensive coverage of long-tail concepts. Additionally, our analysis of the models’ multilingual capabilities reveals performance enhancements in low-resource languages. Notably, we found that applying quality filters, such as those based on CLIP, which are commonly employed to boost performance by reducing pre-training dataset size, can unintentionally diminish the cultural diversity present in large-scale datasets. These findings underscore that while conventional benchmarks may not see substantial benefits from scaling noisy, raw web data to 100 billion examples, this volume of data is essential for developing genuinely inclusive multimodal systems.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC