⚡️ Qwen2.5-1.5B-Nogai-LoRA (Phase 1 Baseline)
Experimental Continuous Pre-Training (CPT) Adapter for Zero-Resource Turkic NLP
A parameter-efficient low-rank adaptation establishing the mathematical baseline for Nogai morphological mapping.
🔬 Model Overview
This is a Quantized LoRA (QLoRA) adapter for Qwen2.5-1.5B-Instruct, engineered entirely on constrained consumer silicon via the Apple mlx-lm framework.
This model represents Phase 1 of the NogaiLLM adaptation pipeline. It was trained exclusively on the unstructured text of the Nogai-Unified-Corpus-v1 (~4 million tokens) for 2,500 iterations. Its primary objective was to evaluate the resilience of dense Byte-Pair Encoding (BPE) architectures to zero-resource Cyrillic vocabulary expansion.
⚠️ Scientific Disclaimer: Catastrophic Forgetting
This adapter is released strictly for academic reproducibility and baseline measurement. It is not a conversational agent.
Because this adapter was trained via Continuous Pre-Training (CPT) on raw, unstructured journalistic text without an accompanying instruction-tuning phase, it exhibits severe Catastrophic Forgetting.
- The Success: The multi-head attention layers successfully mapped the local Turkic morphology, dropping perplexity from a raw 191.13 to an optimized 12.14. The model correctly generates Nogai-specific Cyrillic digraphs (e.g.,
нъ, аь, оь, уь).
- The Failure State: The unstructured training completely overwrote the base model's latent instruction-following manifold. When presented with standard conversational templates (e.g.,
ChatML), the model's predictive routing collapses. It acts solely as an unconditional text generator, aggressively mimicking newspaper formatting rather than answering prompts.
(Note: To test the fully recovered, instruction-following model, please refer to the Phase 2 SFT model: Nogai-SFT-Experimental).
📊 Empirical Benchmarks & Mode Collapse
This adapter was evaluated against Western-centric models in a rigorous 6-model architectural ablation study. The results prove that rigid, Latin-optimized tokenizers (Llama) cannot be force-mapped onto zero-resource Turkic languages via LoRA alone, whereas Eastern architectures (Qwen) achieve successful structural transfer.
| Model Configuration | Scale | Test Loss ($\downarrow$) | Perplexity ($\downarrow$) | Nogai Typographic Density ($\uparrow$) |
|---|
| Qwen-2.5-0.5B (Raw Base) | 0.5B | 5.401 | 221.62 | 0.72% |
| Qwen-2.5-0.5B (Adapted) | 0.5B | 3.379 | 29.33 | 22.95% |
| Qwen-2.5-1.5B (Raw Base) | 1.5B | 5.253 | 191.13 | 17.16% |
| Qwen-2.5-1.5B (Adapted) | 1.5B | 2.497 | 12.14 | 15.05%* |
| Llama-3.2-1B (Raw Base) | 1.0B | 5.836 | 342.44 | 89.66%** |
| Llama-3.2-1B (Adapted) | 1.0B | 5.505 | 245.82 | 0.00% (Mode Collapse) |
*Represents morphologically accurate digraph placement, free of hallucination loops.
**Artifact: Llama base exhibited severe OOD fragmentation, falling into an autoregressive Cyrillic repetition loop.
🚀 Usage (Apple Silicon / MLX)
Because this model acts as an unconditional morphological generator rather than a chat model, you must bypass the chat template during generation.
CLI Execution
Install the MLX framework natively on macOS:
Run generation with the --ignore-chat-template flag to prevent format-induced hallucination:
1mlx_lm.generate \
2 --model Qwen/Qwen2.5-1.5B-Instruct \
3 --adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
4 --prompt "Буьгуьнги куьн" \
5 --max-tokens 128 \
6 --temp 0.5 \
7 --ignore-chat-template
Python API Integration
1from mlx_lm import load, generate
2
3# Load the base model and dynamically apply the Phase 1 LoRA weights
4model, tokenizer = load(
5 "Qwen/Qwen2.5-1.5B-Instruct",
6 adapter_path="ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA"
7)
8
9# Unconditional text completion (Do NOT apply chat templates)
10prompt = "Ногай районынынъ орталыгы Терекли-Мектеб авлында"
11
12response = generate(
13 model,
14 tokenizer,
15 prompt=prompt,
16 max_tokens=100,
17 verbose=True
18)
19
20print(response)
📚 Citation
If you analyze this adapter's weights, utilize the ablation data, or study our mode collapse findings, please cite the core ACM TALLIP / arXiv paper:
1@article{zeinulla2026nogaillm,
2 title={NogaiLLM: Parameter-Efficient Continuous Pre-Training and Architectures of Catastrophic Forgetting in Zero-Resource Turkic Languages},
3 author={Zeinulla, Ansar},
4 journal={arXiv preprint arXiv:2607.xxxxx},
5 year={2026},
6 publisher={Nazarbayev University / ACM TALLIP}
7}
