You can deploy this model and start earning money today!

cyberandy/
sangue-e-grafi-nemotron-nano-grpo

Name: sangue-e-grafi-nemotron-nano-grpo
Availability: InStock

Views

No views yet

🩸 Sangue e Grafi — Nemotron Mini 4B GRPO Adapter

GRPO-reinforced LoRA adapter for Nemotron — the RL stage that aligns tool-use with ontology constraints.

Model Description

This is a LoRA (PEFT) adapter trained via Group Relative Policy Optimization (GRPO) on top of the Nemotron SFT adapter (already merged into nvidia/Nemotron-Mini-4B-Instruct). It is the second training stage for the Nemotron-based pipeline in the Sangue e Grafi project.

Training Pipeline

nvidia/Nemotron-Mini-4B-Instruct (4B)
        │
        ▼
   ┌─────────┐     500 adversarial scenarios
   │   SFT   │◄─── teacher traces (Gemini 2.5 Flash)
   └────┬────┘
        │  merge adapter
        ▼
   ┌─────────┐     reward: correct answer + valid tool use
   │  GRPO   │◄─── ontology constraint checking
   └────┬────┘
        │
        ▼
  THIS ADAPTER (apply on merged SFT base)

Training Details

Parameter	Value
Method	GRPO (Group Relative Policy Optimization)
Base	`nvidia/Nemotron-Mini-4B-Instruct` + merged SFT adapter
Reward signals	Answer correctness + ontology-valid tool usage
Format	SafeTensors LoRA adapter

Benchmark Results 📊

Cross-Architecture Comparison

Model	Hard Dev-Set Accuracy
Gemma 4B (SFT+GRPO)	5/10 (50%)
Nemotron 4B (SFT+GRPO)	4/10 (40%)
Gemini 2.5 Flash (no KG)	3/10 (30%) — easy set

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Step 1: Load base + merge SFT
base = AutoModelForCausalLM.from_pretrained("nvidia/Nemotron-Mini-4B-Instruct")
sft_model = PeftModel.from_pretrained(base, "cyberandy/sangue-e-grafi-nemotron-nano-sft-v7")
merged = sft_model.merge_and_unload()

# Step 2: Apply GRPO adapter
model = PeftModel.from_pretrained(merged, "cyberandy/sangue-e-grafi-nemotron-nano-grpo")
tokenizer = AutoTokenizer.from_pretrained("nvidia/Nemotron-Mini-4B-Instruct")

Intended Uses & Limitations

Intended uses:

Cross-architecture validation of KG-grounded RL reasoning
Research on GRPO reward shaping for tool-use agents
Comparison with Gemma-based GRPO pipeline

Limitations:

Must be applied on top of the merged SFT checkpoint — not standalone
Requires the full KG agent framework for reported performance
Domain-specific to Italian kinship / inheritance law

Project Links

Resource	Link
🚀 Live Demo	HF Space
📦 GitHub	cyberandy/sangue-e-grafi
📄 Paper	RLM-on-KG (arXiv:2604.17056)
🎓 SFT Adapter	sangue-e-grafi-nemotron-nano-sft-v7
📊 Agent Traces Dataset	sangue-e-grafi-agent-traces

Citation

bibtex

@misc{sangue-e-grafi-2026,
  title   = {Sangue e Grafi: Small Models Beat Frontier LLMs on Adversarial Kinship Reasoning with Knowledge Graph Agents},
  author  = {Andrea Volpini},
  year    = {2026},
  url     = {https://github.com/cyberandy/sangue-e-grafi},
  note    = {Hugging Face Build Small Hackathon 2026}
}