Views
No views yet
sorry):| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen2.5-32B-Instruct |
| Method | QLoRA (4-bit NF4, double quantization) |
| LoRA rank | 16 (alpha 32) |
| LoRA targets | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Loss | Buleyean complement KL divergence (sparse) |
| Alpha | 0.7 |
| Training data | Rejection-only (converted from UltraFeedback, chosen discarded) |
| Curriculum | Void curriculum (rejection_density weighting) |
| Steps | 563 |
| Training time | 62 minutes (A100 80GB) |
| Final loss | 0.852 |
| Optimality gap | 1.9% |
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-32B-Instruct", device_map="auto")
5model = PeftModel.from_pretrained(base, "forkjoin-ai/buleyean-qwen2.5-32b")
6model = model.merge_and_unload()
7
8tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-32B-Instruct")1@misc{buley2026buleyean,
2 title={Buleyean Reinforcement Learning: Training from Rejection Alone},
3 author={Taylor Buley},
4 year={2026},
5 url={https://github.com/forkjoin-ai/buleyean-rl}
6}