Title: Accelerating Sample-Efficient Learning in Chinese Language Modeling via Visual Glyphs: A Hot-Start Approach

Abstract: This study investigates whether representing Chinese characters as visual glyph images, instead of the discrete token IDs utilized by current mainstream large language models (LLMs), offers an effective inductive bias for character-level language modeling. Our primary discovery reveals a nuanced outcome: while visual inputs trigger a significant hot-start effect, more than doubling early-stage accuracy during the initial epoch (accounting for just 0.4% of total training steps)—achieving 12.3% compared to the 5.8% baseline of index-based methods—both methodologies ultimately converge to nearly the same final accuracy of 39%. This phenomenon remains consistent across various conditions, including image resolutions as low as 8x8 pixels, partial cropping of up to 50%, and model scales ranging from 110 million to 1.78 billion parameters. We identify the underlying mechanism as the pre-encoding of radical-based structures into the embedding space prior to any training, evidenced by a cosine similarity of 0.27 compared to 0.002 for random embeddings. This pre-encoding facilitates faster alignment but does not enhance final model capacity. Consequently, our findings delineate both the potential benefits and the inherent constraints of employing visual representations as inductive biases in Chinese language modeling.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC