NyayaLM: Qwen3 Legal LoRA Model
Model Card
Model Details
- Base Model:
unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit
- Finetuned By: chhatramani
- License: Apache-2.0
- Language: English & Nepali (Legal domain)
- Frameworks: Unsloth, TRL, Transformers
LoRA Configuration
- LoRA Rank (r): 32
- Target Modules:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- LoRA Alpha: 32
- LoRA Dropout: 0
- Bias: none
- Gradient Checkpointing: enabled (
unsloth)
- Random State: 3407
Dataset
The model was trained on a mixed bilingual legal dataset:
- Legal Datasets:
- Nepal Civil Code (English)
- Muluki Dewani (Nepali)
- General Instruction Datasets (downsampled to 10% of legal size):
- Nepali Alpaca Instruction Dataset
- English Alpaca Instruction Dataset
Final Dataset Size:
- Training samples: ~3145
- Evaluation samples: ~350
Hyperparameters
- Max Sequence Length: 2048
- Batch Size (train): 4
- Batch Size (eval): 2
- Gradient Accumulation Steps: 4
- Warmup Ratio: 0.1
- Learning Rate: 1e-4
- Optimizer: AdamW (8-bit)
- Weight Decay: 0.01
- Scheduler: Cosine
- Epochs: 1
- Seed: 3407
- Evaluation Strategy: every 40 steps
Traning Loss
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\ /| Num examples = 3,145 | Num Epochs = 1 | Total steps = 197
O^O/ _/ \ Batch size per device = 4 | Gradient accumulation steps = 4
\ / Data Parallel GPUs = 1 | Total batch size (4 x 4 x 1) = 16
"-____-" Trainable parameters = 66,060,288 of 4,088,528,384 (1.62% trained)
Unsloth: Will smartly offload gradients to save VRAM!
[197/197 54:51, Epoch 1/1]
| Step | Training Loss | Validation Loss |
|---|
| 40 | 0.749500 | 0.827659 |
| 80 | 0.591000 | 0.742231 |
| 120 | 0.663600 | 0.718418 |
| 160 | 0.632600 | 0.707453 |
Evaluation Metrics
The model was evaluated on a held-out test set of ~350 samples.
Results:
EVALUATION RESULTS
ROUGE-1: 0.5491
ROUGE-2: 0.2151
ROUGE-L: 0.4245
BERTScore F1: 0.9042
Intended Use
NyayaLM is designed as a legal assistant for Nepali law, capable of answering questions, summarizing statutes, and providing structured legal reasoning.
Limitations
- Focused on Nepali and English legal texts;
- LoRA adapters only; full model weights not included.
- Evaluation metrics are based on a small test set and may not generalize.
Uploaded model
- Developed by: chhatramani
- License: apache-2.0
- Finetuned from model : unsloth/qwen3-4b-instruct-2507-unsloth-bnb-4bit
This qwen3 model was trained 2x faster with
Unsloth
