Title: Teaching Arithmetic to Language Models: A Pedagogical Approach

Abstract

This study explores the potential of human mathematics teaching methods to steer the training of language models toward improved arithmetic reasoning. Leveraging GASING, an Indonesian instructional technique that performs basic arithmetic via a left-to-right sequence mirroring the causal flow of token generation, we translate each mathematical operation into a computational process. The execution trace of these processes is then serialized into natural-language Chain-of-Thought (CoT) data for supervision.

Using exclusively a next-token prediction objective, we trained a 86-million-parameter GPT-2 decoder from scratch on this dataset. The model utilizes a syllabic-agglutinative TOBA tokenizer designed for Indonesian and does not rely on reinforcement learning or reward-based optimization. Our analysis of the training trajectory identifies three distinct phases of learning. Through mechanistic investigations—including attention-masking interventions on the CoT information graph, residual-stream probing, and logit-lens inspection—we observed that the model initially encodes a procedural pathway. Subsequently, it cultivates an associative, “mental-arithmetic” ability, allowing it to retrieve intermediate results without relying on explicit step-by-step calculation.

The resulting model achieves greater than 80% accuracy on unseen problems, delivering performance comparable to significantly larger language models. These findings suggest that training grounded in pedagogical principles can produce robust and cost-effective arithmetic capabilities in small-scale models.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC