You can deploy this model and start earning money today!

iqbalahdagita/
Qwen2.5-7B-Indonesian-LoRA

Name: Qwen2.5-7B-Indonesian-LoRA
Availability: InStock

2407.19672

Views

No views yet

Qwen2.5-7B-Indonesian-LoRA

Fine-tuning Qwen2.5-7B-Instruct with LoRA on Indonesian instruction dataset for better Bahasa Indonesia conversational fluency.

Overview

Component	Value
Base Model	`Qwen/Qwen2.5-7B-Instruct` (7.6B params)
Dataset	`cahya/instructions_indonesian` (~52K Indonesian instruction-response pairs)
Method	LoRA SFT (Supervised Fine-Tuning)
LoRA Rank	r=32, alpha=16
Trainable Params	~0.1% of full model
Learning Rate	2.0e-4 (10x base for LoRA)
Epochs	2
Max Seq Length	2048
Effective Batch Size	16 (batch=2 × grad_accum=8)

Training Recipe

The training follows the approach from SeaLLMs-v3 (arXiv:2407.19672) which demonstrated that Qwen2.5 tokenizer has superior Indonesian/SEA language coverage versus Llama/Mistral. Key hyperparameters from published results:

lr=2.0e-4 for LoRA (vs 2.0e-5 for full FT)
warmup_ratio=0.1 with cosine decay
packing=True for training efficiency
bf16 mixed precision
gradient_checkpointing for memory efficiency

Dataset Format

The dataset is preprocessed from the original text/label columns into conversational messages format:

json

{
  "messages": [
    {"role": "user", "content": "Bagaimana cara meningkatkan koneksi pikiran-tubuh saya?"},
    {"role": "assistant", "content": "1. Mulailah dengan mengembangkan latihan kesadaran yang teratur..."}
  ]
}

How to Run

Option 1: HF Jobs (cloud GPU)

Top up credits at https://huggingface.co/settings/billing
Submit via hf_jobs or trl CLI:

bash

python -m hf_jobs.run \
  --script train.py \
  --hardware a10g-large \
  --timeout 7h \
  --dependencies transformers,trl,torch,datasets,peft,trackio,accelerate

Option 2: Local / RunPod / Colab

Requires GPU with ≥24GB VRAM (e.g., A10G, A100, RTX 3090/4090):