Base model: Qwen/Qwen2.5-3B-Instruct Fine-tuning pipeline: SFT → DPO → GRPO Training data: IRC Title 26 (Internal Revenue Code), U.S. Code of Federal Regulations Title 26

This is v3 of the IRS Tax Code RL project.

Training Stages

1. SFT (Supervised Fine-Tuning)

Adapters in: sft/
Grounded on IRC Title 26 question-answer pairs
Multiple checkpoints retained (0000200_adapters.safetensors through 0001000_adapters.safetensors)
Final adapter: sft/adapters.safetensors

2. DPO (Direct Preference Optimization)

Adapters in: dpo/
Trained on preference data generated from SFT model outputs
Best checkpoint: dpo/adapters_best.safetensors

3. GRPO (Group Relative Policy Optimization)

Adapters in: grpo/
RL fine-tuning with citation accuracy and answer correctness reward signals
Best checkpoint: grpo/adapters_best.safetensors

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

# Load the final GRPO adapter (recommended)
model = PeftModel.from_pretrained(base, "dennisonb/qwen25-tax-3b-v3-adapters/grpo")

Intended Use

This model is intended for research and educational purposes related to U.S. tax law (IRC Title 26). It is NOT a substitute for professional tax advice.