Title: TajikNLP: A Comprehensive Open-Source Solution for Processing Tajik Text in Cyrillic Script

Abstract

Publicly accessible natural language processing (NLP) resources for the Tajik language, which is written in the Cyrillic script, are currently scarce. This lack of infrastructure poses significant challenges for both linguistic scholarship and practical application development. To address this gap, we present TajikNLP, an open-source Python library designed to offer the first complete pipeline for processing genuine Tajik text while maintaining its original Cyrillic orthography.

The library is built on a modular architecture anchored by a unified Doc object. This design allows for the sequential execution of various processing components, including text cleaning and normalization, tokenization (with support for subword Byte Pair Encoding), morphemic segmentation, part-of-speech (POS) tagging, stemming, lemmatization, and sentence splitting. A key innovation within the library is its novel unified morphology engine, which provides both controlled and deep analysis modes. This feature markedly enhances the handling of Tajik’s complex agglutinative inflections in both nominal and verbal forms.

Additionally, the release includes a lexicon-based sentiment analyzer and pre-trained Word2Vec and FastText embeddings, which are directly accessible via the Hugging Face Hub. To support reproducibility and encourage further research, we have openly published four linguistic datasets under permissive licenses: a POS-tagged corpus containing 52,500 entries, a sentiment lexicon with 3,500 entries, a toponym gazetteer comprising 5,600 entries, and a personal names dataset with 3,800 entries.

The reliability of TajikNLP is demonstrated through an extensive test suite comprising 616 automated tests, which achieve 93% source code coverage. By establishing this foundational technological infrastructure, TajikNLP lowers the entry barrier for academic and industrial applications, particularly benefiting low-resource environments that utilize Cyrillic script.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC