You can deploy this model and start earning money today!

acv1229/
rl-clarify-orig-prompt-d1-0p5

Name: rl-clarify-orig-prompt-d1-0p5
Availability: InStock

safetensors

reinforcement-learning

Qwen/Qwen2.5-Coder-7B-Instruct

adapter

apache-2.0

Views

No views yet

rl-clarify-orig-prompt-d1-0p5

LoRA fine-tune of Qwen2.5-Coder-7B-Instruct trained with PPO-Lagrangian constrained RL on HumanEvalComm.

PPO-Lagrangian RL fine-tune of Qwen2.5-Coder-7B-Instruct (LoRA rank 16) on HumanEvalComm with question budget d1=0.5. All iter_* checkpoints included. Eval checkpoint: iter_0069.

Training setup

Algorithm: PPO with Lagrangian constraint on avg questions per episode
LoRA rank: 16, alpha 32
Question budget (d1): 0.5
Eval checkpoint: iter_0069
Eval set: 469 problems (full HumanEvalComm eval split, greedy decoding)

Checkpoints

Each iter_XXXX/ folder contains LoRA adapter weights and a log.json with per-iteration training metrics (avg_reward, avg_questions, lambda1).

Training was run in two segments (job preemption on HPC); checkpoints are merged from both segments in chronological order.

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base, "acv1229/rl-clarify-orig-prompt-d1-0p5", subfolder="iter_0069")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")