Title: Tailoring Large Language Models for Bashkir: A Comparative Analysis of LoRA and QLoRA in a Low-Resource Agglutinative Context

Abstract

This study investigates the efficacy of parameter-efficient fine-tuning (PEFT) techniques, specifically Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA), in adapting large language models to Bashkir, a Turkic language characterized by its agglutinative structure and low-resource status. The experimental framework utilizes a Bashkir text corpus comprising 46.9 million tokens across 71,000 documents. Evaluations were performed on a diverse range of model architectures, including DistilGPT2, GPT-2 (in base and medium variants), Phi-2, Qwen2.5-7B, DeepSeek-7B, and Mistral-7B. To mitigate variance and enhance result reliability, every configuration was subjected to training with three distinct random seeds.

The assessment of perplexity on the test set revealed that full fine-tuning of the GPT-2 medium model yielded the lowest score at 3.34. In contrast, QLoRA implementation on Mistral-7B and Phi-2 models achieved competitive performance metrics of 3.79 and 3.81, respectively, while requiring more than 40 times fewer trainable parameters. However, the study also identified instances of substantial performance decline when applying PEFT to specific architectures; for example, DeepSeek-7B with a rank of 8 resulted in a perplexity of 129.55. These findings underscore the critical influence of the base model and its tokenizer on the success of fine-tuning strategies.

Furthermore, a qualitative assessment of outputs generated from Bashkir prompts indicated a divergence between statistical metrics and linguistic coherence. Models exhibiting the lowest perplexity did not always produce the most coherent text. Notably, while the fully fine-tuned model with the best perplexity scores frequently code-switched to English, models fine-tuned with QLoRA consistently generated monolingual Bashkir continuations. The results indicate that applying QLoRA to 7B-scale models presents a viable balance between computational efficiency and output quality for Bashkir. Upon acceptance, the authors commit to releasing the open data, source code, and trained adapters to facilitate reproducibility.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC