You can deploy this model and start earning money today!

Xiazhiqi2002/
icl_best50

Name: icl_best50
Availability: InStock

Views

No views yet

Cross-Image ICL Best-50 Eval Package

Test set for cross-image in-context-learning object detection. 8 classes × 50 query images per class = 400 samples per K, K ∈ {1, 2, 4, 8}.

Layout

cross_icl_best50_package/
├── README.md                  (this file)
├── inputs/
│   ├── inputs_k1.jsonl        (400 samples, K=1)
│   ├── inputs_k2.jsonl        (400 samples, K=2)
│   ├── inputs_k4.jsonl        (400 samples, K=4)
│   └── inputs_k8.jsonl        (400 samples, K=8)
├── support_images/
│   ├── K1/<class>/<stem>__K1.jpg   (red boxes drawn, K=1 subset)
│   ├── K2/<class>/<stem>__K2.jpg
│   ├── K4/<class>/<stem>__K4.jpg
│   └── K8/<class>/<stem>__K8.jpg
└── query_images/
    └── <class>/<file>              (clean, no annotations)

All image paths inside inputs/*.jsonl are package-relative. After untar, they resolve correctly from the package root.

Sample schema (one JSONL line)

field	type	notes
`sample_id`	str	`<class>__<support_stem>__K{n}__<query_stem>`
`class`	str	one of 8 classes
`k`	int	1 / 2 / 4 / 8
`prompt`	str	ready-to-send user text with `<bbox_start>...<bbox_end>` tokens filled
`prompt_template`	str	which of the 2 templates was used
`support_image`	str (rel path)	reference image with K red boxes drawn
`support_image_size`	`[W, H]`	always `[1024, 1024]`
`support_boxes_pixel`	`[[x1,y1,x2,y2], ...]`	K reference boxes in pixel coords
`support_boxes_tokens`	str	same boxes serialized as tokens
`query_image`	str (rel path)	clean query image
`query_image_size`	`[W, H]`	always `[1024, 1024]`
`query_gt_boxes_pixel`	`[[x1,y1,x2,y2], ...]`	ground-truth boxes for evaluation
`query_gt_boxes_tokens`	list[str]	same GT serialized as tokens
`support_stem`	str	support image stem (no extension, no K suffix)

The 2 prompt templates (randomly chosen per sample):

A: "Please detect all objects belonging to the same category as the boxes [{boxes}] in the second image. The boxes are marked in the first image."
B: "The first image shows reference objects marked by red boxes at [{boxes}]. Please detect all objects of the same category in the second image."

How to run inference on another model

For each sample, build a multi-modal user message:

[
  {"type": "image", "path": support_image},
  {"type": "image", "path": query_image},
  {"type": "text",  "text": prompt},
]

Apply your model's chat template, then run inference.

Parsing model output

Boxes are emitted as <bbox_start><xA><yB><xC><yD><bbox_end> with A,B,C,D ∈ [0, 1024]. Extract with regex:

python

import re
TOK = re.compile(r"<x(\d+)><y(\d+)><x(\d+)><y(\d+)>")
def parse(text: str) -> list[list[int]]:
    boxes = []
    for m in TOK.finditer(text or ""):
        x1, y1, x2, y2 = (int(v) for v in m.groups())
        if 0 <= x1 <= 1024 and 0 <= y1 <= 1024 and x2 > x1 and y2 > y1:
            boxes.append([x1, y1, x2, y2])
    return boxes

Since all images are 1024×1024, the grid integers are also pixel coordinates. For non-1024 images: pixel = grid / 1024 * img_size.

Evaluation metrics

Recommended (all per-class micro, then average over classes if you want a single number):

P@50 / R@50 / F1@50 at IoU 0.50 (greedy match)
mF1@50:95 = mean over t ∈ {0.50, 0.55, ..., 0.95} of F1 at t (COCO-style IoU sweep, score-free)
mIoU@50:95 = mean over t of (Σ IoU of TP pairs at t / num total GT)
mIoU_best = mean over GT boxes of max IoU with any pred (threshold-free)

Reference evaluation scripts (from the original Qwen run):

_remote_best50_mf1_5095.py — P/R/F1/mF1
_remote_best50_miou_5095.py — mIoU@50:95 + mIoU_best
_remote_best50_per_class_k8_prf_miou.py — per-class table

To use them with your model's output, write a JSONL with at least these fields per sample:

{"class": ..., "k": ..., "query_gt_boxes_pixel": [...], "pred_boxes_pixel": [...]}

and point the script's BEST50 / pred_k{k}.jsonl to your file.

Reference metrics (Qwen2.5-VL, ckpt_0514_stage2_iter12629)

clean7 (excl. square_cage), micro, K=8:

P@50 = 0.6702
R@50 = 0.8240
F1@50 = 0.7392
mF1@50:95 = 0.4113
mIoU@50:95 = 0.3729
mIoU_best = 0.6365

Per-class @ K=8 (see metrics_per_class_k8_prf_miou.json for full breakdown).

Xiazhiqi2002/icl_best50