Dongxiang–Chinese Translation Project
Project Overview
The Dongxiang–Chinese Translation Project is an open, non-profit initiative dedicated to building high-quality bilingual resources and translation tools for Dongxiang, a low-resource and underrepresented language, with the goal of supporting language preservation, accessibility, and computational research.
Background
Dongxiang (Santa) is a minority language spoken primarily in Gansu Province, China. Due to the scarcity of digitized corpora and linguistic resources, Dongxiang is considered a low-resource language in the field of Natural Language Processing (NLP). This project aims to bridge that gap by systematically constructing bilingual dictionaries, aligned sentence data, and experimental translation systems between Dongxiang and Chinese, while also providing a public-facing platform for cultural and linguistic dissemination.
Key Features
- Dongxiang ↔ Chinese bilingual translation system supporting word-level and sentence-level translation, designed specifically for low-resource language settings;
- Carefully curated Dongxiang–Chinese dictionary, with systematic cleaning, normalization, and disambiguation to improve linguistic clarity and computational usability;
- NLP-oriented dataset design, enabling downstream tasks such as machine translation, alignment, and representation learning;
- Public-facing web platform, providing accessible translation tools while promoting Dongxiang language and cultural awareness;
- Extensible and research-friendly project structure, allowing seamless integration of new data, models, and evaluation strategies.
Project Structure
1.
2├── about.html # Project website
3├── bilingual.html
4├── customs.html
5├── dictionary.html
6├── history.html
7├── index.html
8├── overview.html
9├── status.html
10├── 404.html
11│
12├── photos/ # Images used by the website and documentation
13│
14├── src/ # Source code and data for the translation project
15│ ├── Dongxiang_bilingual_book_process.ipynb # Processes bilingual book data
16│ ├── Dongxiang_dictionary_process_and_analysis.ipynb # Parses and analyzes the raw Dongxiang–Chinese dictionary
17│ ├── Dongxiang_unpublished_data_process.ipynb # Processes unpublished Dongxiang language data
18│ ├── NLLB_training(Chinese___Dongxiang).ipynb # Fine-tunes an NLLB model for Chinese → Dongxiang translation
19│ ├── NLLB_training(Dongxiang___Chinese).ipynb # Fine-tunes an NLLB model for Dongxiang → Chinese translation
20│ ├── Test.ipynb # Testing and exploratory experiments
21│ ├── word_search.ipynb # Converts the raw dictionary into a searchable display
22│ │
23│ └── data/ # Linguistic data resources
24│ ├── Dongxian_Dictionary11.txt # Raw Dongxiang–Chinese dictionary
25│ ├── df_bilingual_book.csv # Processed bilingual book data
26│ ├── df_interior.csv # Processed internal dataset
27│ ├── 东乡语366句会话句 少数民族语汉英日俄对照.txt # Raw bilingual book
28│ ├── 东乡语料文字部分1.txt # Raw internal dataset
29│ ├── 东乡语料文字部分2.txt # Raw internal dataset
30│ └── word_search.csv # CSV derived from Dongxian_Dictionary11.txt for word lookup
31│
32├── README.md # Project overview and usage instructions
33└── LICENSE # Open-source license
Data Description
The Dongxiang–Chinese Translation Project is built on both lexical and sentence-level bilingual data. At the lexical level, a Dongxiang–Chinese dictionary is used to construct a searchable word lookup engine. The dictionary is also used in our EDA process. At the sentence level, bilingual training data is compiled from publicly published materials, as well as internal datasets provided by the Dongxiang County government and Mr. Chen Yuanlong. These sources offer aligned Dongxiang–Chinese sentence pairs suitable for downstream translation modeling. We provide both raw and cleaned datasets to support transparency, reproducibility, and flexible reuse for further experiments and modeling.
Models
We build our translation system on top of NLLB-200-distilled-600M, a multilingual neural machine translation model released by Meta as part of the No Language Left Behind (NLLB) project. NLLB-200 is designed to support translation across more than 200 languages, with a particular emphasis on low-resource and underrepresented languages. This broad multilingual coverage makes the model a practical and principled choice for low-resource translation research.
Our model was fine-tuned using 42,868 Dongxiang–Chinese bilingual sentence pairs, which were processed and cleaned in previous steps. Given the limited size of the training corpus, fine-tuning the baseline NLLB model is more appropriate than introducing aggressive architectural modifications. We used Adafactor as the optimizer during training. Adafactor is a memory-efficient adaptive optimization algorithm. Benefiting from the distilled model size and the use of Adafactor, the full fine-tuning process can be completed within 12 hours on a single NVIDIA A100 GPU, and can also be executed on Google Colab.
All relevant training configurations, hyperparameters, and experiment settings are documented in detail across two NLLB Jupyter notebooks (.ipynb) included in the repository. An early stopping mechanism was initially planned to prevent overfitting; however, due to the limited size and variability of the training data, the validation loss did not exhibit stable or informative behavior. As a result, early stopping was effectively disabled through our hyperparameter settings.
How to use
-
Use the project directly (no setup required):
Visit the GitHub-hosted website:
https://dongxiangtranslationproject.github.io/
The site provides a bilingual translation interface and also includes background explanations on the development status and cultural context of the Dongxiang people.
-
Use the published models on Hugging Face:
Both translation directions are available:
-
Reproduce the full pipeline:
If you want to follow every step in detail, start from those process.ipynb notebooks, then proceed to run the NLLB training workflow. Each step includes detailed comments and explanations to make the pipeline easy to trace and reproduce in our codes. Therefore, no further elaboration will be made here.
-
Tokenizer note (important for training another language):
Dongxiang, in our current data format, is whitespace-delimited (similar to English) and uses the 26 Latin letters. We found that an English tokenizer can already perform tokenization effectively in this setting.
However, if you plan to fine-tune the NLLB model for other languages (especially those without whitespace word boundaries or without a Latin script), you may need to train a dedicated tokenizer as an additional preprocessing step.
Why This Project Matters
This project represents one of the earliest systematic efforts to digitize and computationally model Dongxiang–Chinese bilingual resources. Up till now, Dongxiang is actively used alongside Chinese in bilingual education programs at the local primary school level in the ethnic autonomous county. By digitizing and structuring traditional paper-based Dongxiang–Chinese dictionaries, this project not only preserves valuable linguistic knowledge but also makes it accessible for education, research, and modern NLP applications.
Roadmap
A portion of the bilingual data has undergone automated preprocessing and cleaning. However, during subsequent manual review, we identified a non-trivial number of residual errors, including misalignments and incorrect translations.
While this data has been used internally to improve model generalization during experimentation, it is not released publicly at this stage to ensure overall data quality. Once the cleaning and validation process is complete, we plan to release the curated bilingual corpus to support reproducibility and further research in low-resource language translation.
License and Data Usage
This project is an open, non-profit initiative intended for research, educational, and language preservation purposes only. Unless otherwise stated, the source code and project materials are provided under the MIT License.
Please refer to the LICENSE file for full license text.
Data Usage Statement
All datasets released as part of this project have been reviewed and published with appropriate consent.
The Dongxiang–Chinese linguistic data, including dictionary entries and aligned text resources, are shared with the permission of relevant contributors and are intended to support low-resource language research and cultural preservation.
Acknowledgements
This project would not have been possible without the generous support of institutions and individuals dedicated to the documentation and preservation of the Dongxiang language.
We would like to express our sincere gratitude to:
- The People’s Government of Dongxiang County for its support of Dongxiang language documentation and preservation efforts.
- Mr. Chen Yuanlong (陈元龙), Deputy Director of the United Front Work Department of the Gansu Provincial Party Committee, one of the compilers of the Dongxiang–Chinese Dictionary, who oversaw the collection and provision of relevant internal Dongxiang–Chinese bilingual datasets for this project.
We are grateful to everyone who supports open research, linguistic diversity, and responsible data sharing.
