Wukong is an open-source Vision-Language-Action (VLA) model focused on spatial reasoning for embodied agents. It is designed for robots that must understand, remember, and reason about 3D space from egocentric visual streams — and then act in it.

Named after the Monkey King (Sun Wukong) of Chinese mythology — famed for his piercing "fiery eyes" (火眼金睛) that see through the world — Wukong aims to give embodied agents a comparably sharp sense of space.

What's in This Release (v0.1, T2 stage)

This repository currently hosts Wukong-T2-Adapter: a QLoRA adapter (rank 16, 51.5M trainable params, 206MB) on top of unsloth/Qwen2.5-VL-7B-Instruct, fine-tuned for visual-spatial question answering from egocentric images.

adapter_config.json + adapter_model.safetensors — the trained adapter (final checkpoint, step 49730)
Tokenizer / preprocessor / chat template files — everything needed to load and chat
eval_results.jsonl — per-item evaluation records (question, reference, prediction, score) for full reproducibility

Training setup: 198,919 samples (195,919 VSI-style spatial QA derived from Hypersim + 3,000 programmatic egocentric↔allocentric coordinate-transform QA), 2 epochs, total batch size 8, ~18.5 hours on a single RTX 5090 with Unsloth. Final training loss ≈ 0.19.

Roadmap stages beyond T2 (semantic waypoint prediction, embodiment-specific action adapters, world-action predictive pathway) are under active development and will be released as separate artifacts.

Evaluation

Held-out spatial QA split from our own data pipeline (never seen in training, n=500), scored in VSI-Bench style: exact match for multiple-choice items; MRA (max(0, 1 − |pred−ref|/ref)) for numeric items.

Question type	n	Score
relative_direction_camera	57	0.877
relative_distance_object	130	0.846
relative_size_object	23	0.826
relative_direction_object	108	0.815
absolute_count	38	0.805
absolute_size_object	27	0.758
relative_distance_camera	4	0.750
relative_count	23	0.739
absolute_distance_object	56	0.610
absolute_direction_object	34	0.551
TOTAL	500	0.782

Breakdown by answer format: multiple-choice (n=345) accuracy 0.832; numeric (n=155) MRA 0.671.

Benchmark	What it measures	Status
VSI-style spatial QA (own held-out split)	Visual-spatial intelligence from egocentric images	0.782 (see above)
VSI-Bench (official)	Visual-spatial intelligence from egocentric video	Planned
SPAR-Bench	3D spatial perception and reasoning	Planned
R2R-CE / RxR-CE (VLN-CE, Habitat)	Instruction-following navigation (SR / SPL)	Planned
Real-robot trials (Unitree)	Sim-to-real transfer	Planned

Note: the reported 0.782 is measured on our own held-out split, not the official VSI-Bench; official-benchmark evaluation is on the roadmap.

Usage

python

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Qwen2.5-VL-7B-Instruct",
    load_in_4bit=True,
    max_seq_length=4096,
)
model.load_adapter("FINWHALE/Wukong")   # adapter files at repo root
FastVisionModel.for_inference(model)

A FastAPI serving wrapper and an evaluation script are available in the companion training pipeline (to be linked from this card).

Intended Use

Spatial question answering and measurement estimation from egocentric images
Research on spatial intelligence, world models, and embodied reasoning
A foundation for downstream fine-tuning on specific robot embodiments (navigation adapters on the roadmap)

Limitations

Metric-precision spatial estimation (exact distances, absolute cardinal directions) remains the weakest capability (0.55–0.61) — an open challenge shared by all current VLM/VLA systems.
This release is a spatial-reasoning VLM stage: it does not output motor commands. Closed-loop navigation requires the upcoming waypoint/action adapters.
Long-horizon imagination may drift; closed-loop correction with real observations is required in deployment.
Not intended for safety-critical deployment without additional validation.

Citation

bibtex

@misc{wukong2026,
  title  = {Wukong: An Open Spatial-Reasoning Vision-Language-Action Model},
  author = {FINWHALE},
  year   = {2026},
  url    = {https://huggingface.co/FINWHALE/Wukong}
}

License

Apache 2.0. Please also comply with the licenses of the underlying base models and datasets.