A reasoning-augmented Indonesian-language assistant, fine-tuned from kareem2808/Qwen2.5-1.5B-Legal-ID-Chatbot (SFT) using GRPO (Group Relative Policy Optimization) to enforce explicit <think>...</think> reasoning before every answer, using custom rule-based reward functions rather than a learned reward model.
Method: GRPO via TRL, LoRA (r=16, alpha=16) via Unsloth, merged to 16-bit for release
Language: Indonesian
License: Apache 2.0
Trained on: Free-tier Google Colab, NVIDIA T4 (16GB)
Why GRPO on top of SFT
SFT alone teaches a model to imitate answer style, but it doesn't shape how the model gets there. For a domain like law, where a confident-sounding but ungrounded answer is worse than an admission of uncertainty, I added a GRPO stage to explicitly reward:
Producing visible step-by-step reasoning before answering,
Reasoning that's substantive rather than a token-length shortcut,
Final answers that align with a reference answer, and
Staying in Indonesian rather than code-switching into English.
This is reinforcement learning with rule-based rewards (regex/heuristic scoring functions), not a trained reward model — a deliberate choice to keep the reward signal transparent, fast, and cheap to run on a free-tier GPU.
Note: This is a general-purpose Indonesian instruction dataset, not a legal-specific corpus. Legal domain framing comes from the system prompt and reward shaping (see below) rather than from domain-specific training examples. This is a known limitation — see Limitations.
Each example was mapped into a GRPO-ready ChatML prompt with a fixed system instruction:
Anda adalah asisten hukum AI yang harus menggunakan proses berpikir mendalam sebelum menjawab.
Jawablah pertanyaan hukum menggunakan format terstruktur berikut:
<think>
[Jelaskan proses berpikir, analisis undang-undang, dan rujukan pasal di sini]
</think>
[Berikan kesimpulan atau jawaban akhir hukum Anda di sini]
Reward Functions
Four rule-based reward functions were combined during GRPO training:
Reward function
What it checks
Range
format_reward_func
Correct single use of <think>/</think> tags, proper ordering, non-empty answer after the closing tag; penalizes duplicated/missing tags
-0.5 to 1.0
reasoning_length_reward
Length of the reasoning inside <think> as a proxy for reasoning depth (thresholds at 50 / 200 characters)
0.0 to 1.0
correctness_reward
Jaccard word-overlap similarity between the final answer and the reference output field
0.0 to 1.0
language_reward_func
Penalizes English code-switching in the final answer using Indonesian/English stopword indicators
-0.5 to 1.0
Training Configuration
python
1GRPOConfig(2 num_generations =4,# group size, reduced from 8 for T4 VRAM limits3 per_device_train_batch_size =1,4 gradient_accumulation_steps =8,5 max_completion_length =512,6 learning_rate =1e-6,7 optim ="adamw_8bit",8 weight_decay =0.01,9 warmup_ratio =0.1,10 lr_scheduler_type ="cosine",11 beta =0.04,# KL penalty vs. the SFT reference model12 max_steps =100,13 fp16 =True,14)
LoRA adapter: r=16, alpha=16, dropout=0, applied to all attention and MLP projections (q/k/v/o_proj, gate/up/down_proj), trained with Unsloth's gradient checkpointing to fit T4 VRAM.
Compute budget note: 100 GRPO steps on a free-tier T4 is a small-scale run intended as a proof of concept for the SFT→GRPO pipeline, not an exhaustively converged policy.
Example Output
Test prompt: "Saya staf admin, kemarin lembur 3 jam untuk beresin laporan. Apakah saya berhak dapat uang lembur?"
<think>
Pertama, mari kita lihat apakah ada peraturan khusus tentang lemburan bagi staf
administrasi. Karena tidak ada undang-undang tertentu secara spesifik mengenai
ini, kita perlu melihat kebijakan perusahaan atau organisasi mana yang
memungkinkan lemburan bagi staf administrasi...
</think>
Sebagai AI, saya tidak memiliki akses ke informasi pribadi atau data perusahaan
Anda, jadi saya tidak dapat memberi Anda jawaban akurat tentang apakah Anda
berhak menerima uang lemburan atau tidak. Penting untuk berkonsultasi dengan
manajemen Anda atau pengusaha Anda...
The model reliably produces visible reasoning before answering and appropriately declines to give a definitive legal conclusion without more context — a reasonable behavior for a legal-adjacent assistant, though it does not cite specific statutes in this example, which reflects the general-purpose (non-legal-specific) training data.
Limitations
Training data is not legal-domain-specific. The model was not trained on Indonesian statutes, case law, or legal QA pairs — its "legal" behavior is shaped by the system prompt and reward functions, not by exposure to legal source text. It should not be trusted to cite accurate articles/pasal or provide legal conclusions without verification against primary sources.
Small-scale GRPO run. 100 steps, group size 4, on a single T4 — sufficient to demonstrate the reasoning-format objective works, not a fully converged RL policy.
correctness_reward uses lexical (Jaccard) overlap, not semantic similarity — it can reward answers that share vocabulary with the reference without being semantically correct, or penalize correct answers phrased differently.
Not evaluated on a held-out legal benchmark. No formal accuracy/factuality evaluation has been run against known-correct legal answers.
Intended Use
Proof-of-concept for combining SFT with rule-based-reward GRPO to elicit structured reasoning in a small (1.5B) Indonesian-language model. Suitable for demonstrating the RLHF/GRPO pipeline and reasoning-format control; not suitable as a standalone source of legal advice.
Training Framework
Trained 2x faster using Unsloth and Hugging Face's TRL library.
