Title: Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing

Abstract

The high cost of input tokens remains a significant bottleneck for AI-assisted coding agents. This overhead is largely driven by two specific issues inherent in raw human input: the inefficiency of tokenizing non-English text and the structural entropy found within conversational prompts. Current solutions tend to be reactive, either compressing contexts that have already become bloated or intervening only after failures have occurred. To address this, we propose a proactive, edge-side middleware for prompt rewriting situated between the developer and the cloud-based agent. This system utilizes a local Llama 3.2 (3B) model to perform cross-lingual translation into English and structurally rewrite inputs into a compact, task-oriented format. To guarantee that the optimized prompt does not exceed the size of the original, we implement regex-validated rewrite-with-fallback safeguards.

We evaluated our approach using OMH-Polyglot, a multilingual coding benchmark that includes Turkish, Arabic, Chinese, and code-switched specifications. Testing across three commercial LLM backends revealed that the middleware reduces prompt tokens by 34–47% and lowers total token usage by up to 18.8%, all while maintaining or enhancing task accuracy. Ablation studies indicate that these improvements stem primarily from the rewriting stage rather than mere function-name extraction. Furthermore, when compared to LLMLingua-2 at equivalent compression rates, our method consistently delivers superior OckScore performance across all tested backends. These findings demonstrate that proactive prompt optimization can significantly cut inference costs without compromising coding quality.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC