Srstilm-Indic-4B-CPT
Indic QLoRA continued-pretrain of Qwen3.5-4B-Base with a +29K-token Indic tokenizer extension (277K vocab). Trained ~13M tokens on a 400K-sequence Indic+code+English corpus (50/30/20); embeddings frozen, LoRA on attn/MLP. On held-out Indic text it lowers loss 9.16 to 6.96 (24%) vs the extended-vocab base. Cost-effective experimental build. Use with the extended tokenizer in this repo.
Model
- License: apache-2.0
- Method: continued pretraining with an extended tokenizer (+Indic merges), Sailor new-row embedding init, clean base-row freeze.
- Languages: hi, mr, ta, te, kn, sa, en
Evaluation
| Benchmark | Score |
|---|
| Held-out Indic loss (base + extended tokenizer) | 9.16 |
| Held-out Indic loss (this model, + LoRA) | 6.96 |
| Held-out Indic loss reduction (%) | 24 |
Intended use & limitations
Research / Indic + code generation. As a continued-pretrained base it inherits Qwen/Qwen3.5-4B-Base's strengths and biases; evaluate before production use.
