⚡️ Qwen2.5-1.5B-Nogai-SFT (Phase 2 Recovery)
Curing Catastrophic Forgetting & Restoring Instruction Alignment in Zero-Resource NLP
An empirical Supervised Fine-Tuning (SFT) framework demonstrating the recovery of ChatML prompt-adherence following aggressive continuous pre-training on morphological isolates.
🔬 Model Overview
This is the Phase 2 Supervised Fine-Tuning (SFT) adapter for Qwen2.5-1.5B-Instruct, engineered as part of the NogaiLLM adaptation pipeline.
During Phase 1 (Continuous Pre-Training), the base model successfully learned the Nogai Cyrillic morphology (dropping Perplexity to 12.14) but suffered from severe Catastrophic Forgetting—it completely lost its latent instruction-following manifold and defaulted to unconditional text generation.
This Phase 2 adapter cures that architectural collapse. Fine-tuned on the highly structured Nogai-Russian-SFT-Biblical-v1 micro-dataset using exact sentence-boundary proportional chunking, this adapter successfully maps the new Nogai morphological weights back onto the model's native ChatML multi-head attention routing.
💻 Hardware Constraints & The "Metal Descriptor Leak"
Like the Phase 1 model, this SFT alignment was executed natively on consumer hardware: an Apple MacBook Pro (M2 Pro, 16GB Unified Memory) utilizing the mlx-lm framework.
Overcoming Apple Silicon Bottlenecks:
Extended multi-epoch SFT introduces a severe hardware-level bottleneck during backpropagation. The macOS Metal driver accumulates the count of active buffer handles (descriptors) faster than the Python runtime can garbage collect, eventually triggering an execution panic (0000000e:Internal Error).
We resolved this by injecting an active cache-clearing protocol (--clear-cache-threshold 0.7) into the MLX allocator. This synchronously flushed Metal command buffers dynamically, successfully bottlenecking peak unified memory to exactly 5.734 GB and allowing the hardware to complete 2,400 continuous iterations (2 full epochs) with absolute systemic stability.
⚠️ Scientific Limitation: "Structural-Semantic Overfitting"
We transparently define the data-ceiling of this adapter. This model is released as a structural baseline proving that parameter-efficient multi-epoch tuning can recover format compliance in zero-resource LLMs.
- The Success: The model flawlessly adheres to
ChatML system prompts, identifies user boundaries, generates cross-lingual translations, and terminates gracefully via <|im_end|> tokens without EOS mode collapse.
- The Failure State (Associative Hallucination): Because the SFT micro-dataset ($N \approx 1,200$) is strictly bound to a religious/narrative domain, the model exhibits Structural-Semantic Overfitting. When prompted with out-of-distribution (OOD) modern text (e.g., medical or technological queries), the predictive routing maps to the nearest active high-probability cluster, frequently surfacing hallucinated biblical terminology (e.g., "grace", "brethren") rather than literal translations.
Achieving open-domain zero-shot conversational fluency requires scaling this baseline via Synthetic Back-Translation.
🚀 Architectural Dependency & Usage (Crucial)
Because this Phase 2 SFT adapter relies on the morphological vocabulary learned in Phase 1, it cannot be applied directly to the raw Qwen 2.5 model.
You must execute a Two-Step LoRA Fusion Pipeline:
- Fuse Phase 1 into the raw base.
- Run inference using Phase 2 on the fused base.
Step-by-Step CLI Execution (Apple Silicon / MLX)
1. Install MLX:
2. Fuse Phase 1 (The Vocabulary Base):
This command downloads the raw Qwen model, applies our Phase 1 morphological weights, and saves the fused model locally.
1mlx_lm.fuse \
2 --model Qwen/Qwen2.5-1.5B-Instruct \
3 --adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
4 --save-path ./local_qwen_1.5B_Nogai_Base
3. Run the Phase 2 SFT Chat Interface:
Now, spin up the interactive chat template by loading the fused base and dynamically applying this Phase 2 SFT adapter.
1python -m mlx_lm.chat \
2 --model ./local_qwen_1.5B_Nogai_Base \
3 --adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-SFT-Experimental \
4 --temp 0.3
Interactive Test Prompt:
Once the chat initializes, paste the following prompt to verify cross-lingual semantic alignment:
Переведи этот текст на русский язык: Бизикилер, яшавлары пайдасыз болмасын деп, оьзлерин ийгиликлер этпеге багысласынлар.
📚 Citation
If you utilize our two-stage adaptation methodology, sentence-boundary chunking, or memory-leak resolutions, please cite the core ACM TALLIP / arXiv paper:
1@article{zeinulla2026nogaillm,
2 title={NogaiLLM: Parameter-Efficient Continuous Pre-Training and Architectures of Catastrophic Forgetting in Zero-Resource Turkic Languages},
3 author={Zeinulla, Ansar},
4 journal={arXiv preprint arXiv:2607.xxxxx},
5 year={2026},
6 publisher={Nazarbayev University / ACM TALLIP}
7}
