You can deploy this model and start earning money today!

bilalfaye/
flan-t5-large-rpo

Name: flan-t5-large-rpo
Availability: InStock

safetensors

google/flan-t5-large

finetune

Views

No views yet

🧠 Flan-T5-{Small|Large|XL}-RPO

🔬 Fine-tuned with Reward Partitioning Optimization (RPO) — a value-free, stable method for single-trajectory reinforcement learning with scalar feedback.

📌 Model Summary

This model is a fine-tuned variant of the Flan-T5 {Small|Large|XL} checkpoint, trained using Reward Partitioning Optimization (RPO). RPO is a new method designed for learning from single-trajectory scalar feedback (e.g., thumbs up/down), and eliminates the need for learning value functions or preference pairs.

✅ Trained with only (prompt, response, reward) triplets.
🔁 No joint optimization, no auxiliary models.
🚀 Efficient and stable training.
🤖 Strong preference alignment (evaluated by LLM-as-a-judge).
📊 Outperforms KTO and DRO in automatic metrics and LLM preference winrate.

🧪 Training Details

Base Model: flan-t5-{small|large|xl}
Dataset: UltraFeedback — high-quality (prompt, response, reward) triplets with multiple completions per prompt.
Feedback Format: scalar reward (e.g., [prompt, response, reward]).
GPU Used: 1× A100 (80GB)
Training Objective: RPO supervised learning using partitioned reward normalization.
Baselines Compared: DRO and KTO.

🤖 Inference

python

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_name = "bilalfaye/flan-t5-{small|large|xl}-rpo"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)

prompt = "How can I improve my productivity working from home?"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.2,
    no_repeat_ngram_size=3,
)

response = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
print(response)

📈 Evaluation Summary

Judge	Win Rate vs DRO	Win Rate vs KTO	Win Rate vs SFT
Mistral	✅ 83–93%	✅ 82–93%	✅ 82–84%
LLaMA	✅ 67–74%	✅ 65–72%	✅ 63–73%

✅ Use Cases

Aligned conversational agents
Helpful, non-toxic instruction following
Scalar feedback training pipelines
Preference-optimized generation (without pairwise preference labels)

📚 Citation

If you use this model, please cite the following paper:

bibtex

@article{faye2025rpo,
  title     = {Value-Free Policy Optimization via Reward Partitioning},
  author    = {Bilal Faye and Hanane Azzag and Mustapha Lebbah},
  journal   = {arXiv preprint arXiv:2406.XXXX},
  year      = {2025}
}

🔗 Related Models

bilalfaye/flan-t5-small-rpo
bilalfaye/flan-t5-large-rpo
bilalfaye/flan-t5-xl-rpo

bilalfaye/flan-t5-large-rpo