mkdir -p checkpoints/UnoGrasp-Ratio-RL-IoU-nlp-small

hf download FBK-TeV/UnoGrasp-Ratio-RL-IoU-nlp-small \
  --local-dir checkpoints/UnoGrasp-Ratio-RL-IoU-nlp-small

If the Hugging Face CLI is not installed:

pip install -U huggingface_hub

Expected local layout:

text

UnoGrasp/
`-- checkpoints/
    `-- UnoGrasp-Ratio-RL-IoU-nlp-small/
        |-- config.json
        |-- model.safetensors.index.json
        |-- model-0000x-of-0000x.safetensors
        |-- tokenizer.json
        `-- ...

Model Description

UnoGrasp-Ratio-RL-IoU-nlp-small is a released UnoGrasp checkpoint for natural-language (NLP) obstruction reasoning on the synthetic small split of UnoBench.

Ratio means the model uses obstruction ratio as obstruction information. RL-IoU means the model uses RFT (GRPO) with a set-level IoU reward. nlp indicates the natural-language setting, and small indicates that the model was trained on the small subset.

Base Architecture: Qwen2.5-VL (3B)

Training Paradigm:

Supervised Fine-Tuning on UnoBench Train-small
Group Relative Policy Optimization (GRPO)-style RFT on UnoBench Train-small

Usage

Clone or enter the UnoGrasp repository, install the dependencies described there, and make sure the UnoBench synthetic data is available at:

../UnoBench/UnoBenchSyn

You can also load this model directly with Hugging Face Transformers for single-sample inference:

python

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

model_id = "FBK-TeV/UnoGrasp-Ratio-RL-IoU-nlp-small"

# You can also use a local path after downloading:
# model_id = "checkpoints/UnoGrasp-Ratio-RL-IoU-nlp-small"

# We recommend enabling flash_attention_2 for better acceleration and memory saving.
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="auto",
)

processor = AutoProcessor.from_pretrained(model_id)
processor.chat_template = processor.tokenizer.chat_template

# Optional: control the visual token range to balance performance and cost.
# min_pixels = 256 * 28 * 28
# max_pixels = 1280 * 28 * 28
# processor = AutoProcessor.from_pretrained(
#     model_id,
#     min_pixels=min_pixels,
#     max_pixels=max_pixels,
# )

system_prompt = (
    "You are an assistant specialized in robotic grasp planning based on occlusion reasoning. "
    "When asked which object must be removed first to grasp a specific target object in a single image:\n\n"
    "- If the target object is not occluded, return the target object's name/description and its coordinates.\n"
    "- If the target object has one occlusion path, reason step-by-step along that path, include the occlusion "
    "ratio for each occlusion relation when available, and end with the top-most object that must be removed "
    "first. Each reasoning step must reference objects with explicit (x,y) coordinates.\n"
    "- If the target object has multiple occlusion paths, reason step by step for each path separately, and "
    "include all distinct top-most occluding objects in the final answer.\n"
    "- All reasoning must be enclosed within a single pair of <think>...</think> tags.\n"
    "- The final answer must be enclosed in <answer>...</answer> tags and formatted strictly as:\n"
    "  <answer>[<points x y>object name</points>, ...]</answer>\n"
)

messages = [
    {
        "role": "system",
        "content": system_prompt,
    },
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "path/to/rgb_image.png",
            },
            {
                "type": "text",
                "text": "Which object must be removed first to grasp the red mug?",
            },
        ],
    },
]

# Preparation for inference
text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to(model.device)

# Inference
with torch.inference_mode():
    generated_ids = model.generate(**inputs, max_new_tokens=1500)
    generated_ids_trimmed = [
        out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
    ]
    output_text = processor.batch_decode(
        generated_ids_trimmed,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )

print(output_text[0])

Run NLP inference:

bash

python unograsp/inference/infer_nlp.py \
  --model_path checkpoints/UnoGrasp-Ratio-RL-IoU-nlp-small \
  --dataset_json ../UnoBench/UnoBenchSyn/test_nlp_small.jsonl \
  --dataset_root ../UnoBench/UnoBenchSyn \
  --out_dir outputs/inference/nlp_synthetic \
  --max_new_tokens 1500

Evaluate the predictions:

bash

python unograsp/evaluation/evaluate_nlp.py \
  --pred_path outputs/inference/nlp_synthetic/predictions.jsonl \
  --gt_path ../UnoBench/UnoBenchSyn/test_GT_small.json \
  --npz_root ../UnoBench/UnoBenchSyn/annotations \
  --dataset_type synthetic

Inference outputs are written to:

outputs/inference/nlp_synthetic/predictions.jsonl

If this file already exists, inference resumes from unfinished samples.

Dataset

This checkpoint is intended for use with UnoBench, especially the released synthetic small split:

text

test_nlp_small.jsonl
test_GT_small.json
annotations/

Citation

If you use this checkpoint, please cite:

bibtex

@article{jiao2025obstruction,
  title = {Obstruction reasoning for robotic grasping},
  author = {Runyu Jiao and Matteo Bortolon and Francesco Giuliari and Alice Fasoli and Sergio Povoli and Guofeng Mei and Yiming Wang and Fabio Poiesi},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2026}
}