Note : After testing I find that it hallucinate so badly that I can't even recommend anyone using this model. I promised that further model release will be better quality.

Model Description

Base Model: ThaiLLM/ThaiLLM-8B (Qwen3 architecture)
Training Data: Thai legal documents from iapp/rag_thai_laws
Language: Thai (th)
License: Apache 2.0
Training Method: QLoRA (4-bit quantization)

Training Details

Dataset

This model was fine-tuned exclusively on:

iapp/rag_thai_laws (42,755 samples)
- Thai law texts from the Office of the Council of State
- Cleaned and processed dataset
- Contains various types of Thai legal documents

Hardware Configuration

GPU: 2x NVIDIA RTX PRO 4000 (24GB VRAM each)
Training Time: ~2 hours
Framework: PyTorch with Unsloth optimizations

Training Hyperparameters

Method: QLoRA (4-bit quantization)
Max Sequence Length: 2048
Batch Size: 4 per device
Gradient Accumulation Steps: 8 (effective batch size: 64)
LoRA Rank (r): 16
LoRA Alpha: 32
LoRA Dropout: 0.05
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Learning Rate: 2e-4
Embedding Learning Rate: 1e-5
Epochs: 1
Optimizer: adamw_8bit
Warmup Ratio: 0.05
Data Packing: Enabled (2-3x speedup)
Gradient Checkpointing: Enabled

Training Procedure

The model was trained using Unsloth's efficient QLoRA implementation with the following optimizations:

4-bit quantization for memory efficiency
Data packing for faster training on short texts
Gradient checkpointing to reduce memory usage
ChatML template for instruction-following format

Usage

Basic Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "sthaps/ThaiLLM-8B-ThaiLaw"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Example usage
messages = [
    {"role": "system", "content": "คุณเป็นผู้ช่วยด้านกฎหมายไทยที่เชี่ยวชาญ คุณต้องตอบคำถามเกี่ยวกับกฎหมายไทยอย่างถูกต้องและครบถ้วน"},
    {"role": "user", "content": "อธิบายเกี่ยวกับพระราชบัญญัติ"},
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.95)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Using with Unsloth (Recommended)

python

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="sthaps/ThaiLLM-8B-ThaiLaw",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

# Enable faster inference
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "คุณเป็นผู้ช่วยด้านกฎหมายไทยที่เชี่ยวชาญ"},
    {"role": "user", "content": "อธิบายเกี่ยวกับกฎหมายแรงงานไทย"},
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Limitations

Training Data Scope: The model was trained on a single dataset (iapp/rag_thai_laws) and may not represent all areas of Thai law
Accuracy: While trained on legal documents, this is an AI model and may produce incorrect information
Professional Advice: This model should not be used as a substitute for professional legal advice
Language: Primarily optimized for Thai language legal documents
Updates: The model does not have access to real-time legal updates or changes

Model Architecture

Base Architecture: ThaiLLM (8B parameters)
Fine-tuning Method: QLoRA (Low-Rank Adaptation with 4-bit quantization)
Trainable Parameters: ~0.5% of total parameters (LoRA adapters only)
Total Parameters: ~8 billion

Training Metrics

Training Samples: 42,755
Training Steps: ~1,336 (with packing)
Effective Batch Size: 64
Training Loss: Decreased steadily throughout training
Convergence: Achieved in 1 epoch with packing

Acknowledgments

Base Model: ThaiLLM/ThaiLLM-8B by ThaiLLM team
Dataset: iapp/rag_thai_laws by iAPP
Training Framework: Unsloth for efficient fine-tuning