You can deploy this model and start earning money today!

julyanghar/
tosc-model-weights

Name: tosc-model-weights
Availability: InStock

Views

No views yet

TOSC — Triadic Object-State Consistency

Benchmark, evaluation code, and trained LoRA models for the Triadic Object-State Consistency (TOSC) benchmark.

TOSC tests whether a vision-language model keeps its object descriptions consistent across three versions of the same scene: the original image, the same image with one object removed (inpainted out), and the same image with that object replaced by a different one. A consistent model should mention the object when present, stop mentioning it once removed, and report the new object after replacement.

This repository is self-contained: the benchmark images, the evaluation code, and the model adapters are all here, and the evaluation pipeline runs with no external code dependency.

Repository layout

tosc-model-weights/
├── benchmark/                 # 5,969 triplets (origin / removed / replaced)
│   ├── original/images/       #   {N}.jpg
│   ├── repair/images/         #   {N}_masked_{obj}_repair.png        (removed)
│   ├── insert/images/         #   {N}_masked_{obj}_insert_{new}.png  (replaced)
│   ├── TOSC_dataset.jsonl      # generated index (regenerable)
│   └── insertions.jsonl        # generated object metadata (regenerable)
├── eval/                      # evaluation code (see eval/README.md for details)
│   ├── build_index.py         #   image folders -> the two .jsonl files
│   ├── run_eval.sh            #   end-to-end runner: caption -> merge -> score
│   ├── eval_tosc.py           #   scoring engine
│   └── eval_masked_obj_generative.py
├── llava/                     # vendored inference package (imports as `llava`)
├── lora/                      # 4 trained LoRA adapters (TSA-DPO)
└── requirements.txt

Models

Adapter (`lora/…`)	Base model
`LLaVA_v1_5_7b-TSA-DPO`	`liuhaotian/llava-v1.5-7b`
`LLaVA_v1_5_13b-TSA-DPO`	`liuhaotian/llava-v1.5-13b`
`Qwen2_VL_7B-TSA-DPO`	`Qwen/Qwen2-VL-7B-Instruct`
`Qwen2_5_VL_7B-TSA-DPO`	`Qwen/Qwen2.5-VL-7B-Instruct`

Installation

A CUDA GPU is required for caption generation. Python 3.10+ is recommended.

bash

# 1. Get the repo (images + adapters are stored with Git LFS)
git lfs install
git clone https://huggingface.co/julyanghar/tosc-model-weights
cd tosc-model-weights

# 2. Create an environment and install dependencies
conda create -n tosc python=3.10 -y && conda activate tosc   # or use a venv
pip install -r requirements.txt

# 3. (Qwen models only) install flash-attn for the Qwen2-VL / Qwen2.5-VL path,
#    or change attn_implementation to "sdpa" in llava/eval/utils/hf_utils.py
pip install flash-attn --no-build-isolation

Base models (liuhaotian/llava-v1.5-*, Qwen/Qwen2*-VL-*-Instruct) and the CLIP vision tower are downloaded automatically from the Hugging Face Hub on first use; the LoRA adapters are loaded from lora/ in this repo.

How to run

bash

# 1. Build the index files once (regenerable from the images at any time)
python eval/build_index.py

# 2. Evaluate a model. Pick one of the four presets in eval/run_eval.sh
#    (uncomment it), or select it via environment variables. Example: Qwen2.5-VL
#    on 4 GPUs.
GPU_LIST=0,1,2,3 \
OUTPUT_NAME=Qwen2_5_VL_7B-TSA-DPO \
MODEL_NAME=Qwen/Qwen2.5-VL-7B-Instruct \
LORA_NAME=$PWD/lora/Qwen2_5_VL_7B-TSA-DPO \
bash eval/run_eval.sh

run_eval.sh runs three steps: (1) generate a caption for every benchmark image (multi-GPU, sharded by GPU_LIST), (2) merge the shards, (3) score with eval/eval_tosc.py. The inference path is selected automatically — model names containing llava + 1.5 use the LLaVA-1.5 loader, everything else uses the Qwen path. Set LORA_NAME="" to evaluate a base model without the adapter.

Single-GPU / quick smoke test

GPU_LIST=0 bash eval/run_eval.sh # single GPU (slower)

To sanity-check the wiring on a few images, point QUESTION_FILE at a trimmed copy of benchmark/TOSC_dataset.jsonl.

Outputs

Results go to ${TOSC_RESULTS:-/home/yilin/tmp/tosc_results} (outside the repo by default; override with OUTPUT_DIR):

answers/<OUTPUT_NAME>.jsonl — merged captions
<OUTPUT_NAME>_eval.json — full per-triplet results
<OUTPUT_NAME>_eval_summary.json — aggregate metrics

Configuration (environment variables)

Variable	Default	Meaning
`GPU_LIST`	`0`	comma-separated GPU ids, e.g. `0,1,2,3`
`MODEL_NAME` / `LORA_NAME` / `OUTPUT_NAME`	Qwen2.5 preset	model selection
`MAX_NEW_TOKENS`	`512`	caption length
`TEMPERATURE`	`0`	`0` = greedy / deterministic
`SEED`	`42`	random seed
`OUTPUT_DIR`	`/home/yilin/tmp/tosc_results`	where results are written

Metrics

The scorer reports the headline TOSC score plus OPA (sees the object), RCA (drops it after removal), RUA (sees the replacement, not the old object), and fine-grained failure rates (OldPersist / RepFail / CC / MixConf). Object mentions are detected with a local COCO synonym table and lemmatization — no external API or LLM judge. See eval/README.md for the exact formulas and per-state definitions.