PEFT LoRA adapters for a Bangla (Bengali) instruction-tuned Qwen3-4B-Base. Load these on top of Qwen/Qwen3-4B-Base to get the Bangla instruction-following model.

This is SFT-only (no continual pre-training stage). The adapters were produced by supervised fine-tuning on the ihumaunkabir/alpaca-gpt4-bangla instruction dataset, using Unsloth.

Looking for a ready-to-run model? The merged + quantized GGUF (no base model needed) lives at ihumaunkabir/qwen3_bangla_q4_k_m_gguf.

detail	value
Adapter size	~1.2 GB
Base model (required)	`Qwen/Qwen3-4B-Base` -- loaded separately, not bundled
Training precision	QLoRA 4-bit (adapter trained with the base in 4-bit)
LoRA rank	128
LoRA alpha	32
rsLoRA	enabled (`use_rslora=True`)
LoRA targets	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, embed_tokens, lm_head`

embed_tokens and lm_head are included to help the model adapt its vocabulary to Bengali script (the base Qwen3-4B model has a ~152K multilingual vocabulary, but the Bangla portion benefits from SFT-time adaptation).

Training

Stage 1 -- Supervised Fine-Tuning (SFT)

setting	value
SFT data	`ihumaunkabir/alpaca-gpt4-bangla` -- 49,969 Bangla instruction-response pairs
Data lineage	English alpaca-gpt4 -> Korean (FreedomIntelligence/alpaca-gpt4-korean) -> Bangla
Max steps	120 (template smoke test -- set `num_train_epochs=1` for real use)
Learning rate	5e-5 (LoRA), 1e-5 (embeddings)
Packing	disabled

How to use

Load adapters with Unsloth (recommended, 2x faster inference)

python

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "ihumaunkabir/qwen3_bangla_lora",
    max_seq_length = 2048,
    load_in_4bit = True,
)
FastModel.for_inference(model)

alpaca_prompt = """নিচে একটি নির্দেশনা দেওয়া আছে, যা একটি কাজের বর্ণনা দেয়। অনুরোধটি যথাযথভাবে সম্পূর্ণ করে একটি উত্তর লিখুন।

### নির্দেশনা:
{}

### উত্তর:
{}"""

inputs = tokenizer([alpaca_prompt.format("বাংলাদেশের রাজধানীর নাম লেখো।", "")], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128, use_cache=True)
print(tokenizer.batch_decode(outputs))

Note: this uses Unsloth's FastModel API (not FastLanguageModel). FastModel is the unified entry point for newer model families including Qwen3.

Load with PEFT (no Unsloth)

python

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained(
    "ihumaunkabir/qwen3_bangla_lora",
    load_in_4bit = True,
)
tokenizer = AutoTokenizer.from_pretrained("ihumaunkabir/qwen3_bangla_lora")

Limitations

Smoke-test training volume (max_steps=120) -- ~1,920 samples. Increase for production.
No safety alignment. No RLHF, DPO, or red-teaming.
QLoRA 4-bit during training can slightly degrade quality vs. bf16.
Requires the base model to be loaded separately -- this is an adapter, not a standalone model.
Alpaca prompt format required. The model will produce poor output if prompted with a different template (e.g. ChatML).

Citation

If you use these adapters, please cite both this repo and the alpaca-gpt4-bangla dataset, plus the base Qwen3 model.

This repo (LoRA adapters)

bibtex

@misc{qwen3-bangla-lora,
  author       = {ihumaunkabir},
  title        = {qwen3-bangla: Bangla SFT LoRA adapters for Qwen3-4B-Base},
  year         = {2026},
  url          = {https://huggingface.co/ihumaunkabir/qwen3_bangla_lora},
  note         = {SFT on alpaca-gpt4-bangla, trained with Unsloth}
}

SFT dataset -- alpaca-gpt4-bangla

bibtex

@misc{alpaca-gpt4-bangla,
  author       = {ihumaunkabir},
  title        = {alpaca-gpt4-bangla: A Bangla instruction-following dataset},
  year         = {2026},
  url          = {https://huggingface.co/datasets/ihumaunkabir/alpaca-gpt4-bangla},
  note         = {Machine translation (Korean -> Bangla) of FreedomIntelligence/alpaca-gpt4-korean}
}

Base model

bibtex

@misc{qwen3,
  author       = {Qwen Team},
  title        = {Qwen3-4B-Base},
  year         = {2025},
  url          = {https://huggingface.co/Qwen/Qwen3-4B-Base}
}