Capra-v3 is a vision-language navigation (VLN) policy built on Qwen3-VL-4B-Instruct. Given a navigation instruction and a short history of egocentric RGB observations, it predicts navigation as plain text: a pixel-space goal in the current image plus a facing direction, a turn, or a stop. It is a stock Qwen3VLForConditionalGeneration (no custom modeling code, no vocab extension), so it loads with stock transformers and runs with a plain generate() call.

This repo contains the v3 checkpoint (n8, 1 epoch). Unlike the trajectory-token line (CapraXL, which emits delta-bin trajectory tokens), Capra-v3 emits the v3 pixel-goal text contract — a single "u v yaw" waypoint, turn arrows, or STOP.

Results

R2R val_unseen (VLN-CE, 1,839 episodes, discrete Habitat action space, 110° HFOV, greedy decoding):

Metric	Value
SR ↑	0.581
SPL ↑	0.535
OS ↑	0.695
NE ↓	4.25

Model interface

Output contract (text)

Per query the model outputs one of:

output	meaning
`"u v yaw"`	waypoint: `u`/`v` are coord-normalized to 0–1000 of the sensor (denormalize by sensor W/H); `yaw` is an integer heading delta in degrees, wrapped to `[-180, 180]`
`↑` / `←` / `→` (repeated)	discrete turn/step run (`↑` forward, `←` left, `→` right)
`STOP`	task complete

⚠️ Yaw sign. The prompt text says "positive = right", but the model was trained on labels where positive = LEFT (the values came straight from yaw_goal − yaw_cur, no negation). Execute yaw with positive = left.

Parse precedence (mirror the training/eval contract exactly): arrows first (turn outputs never contain digits/STOP), then STOP anywhere in the uppercased text, then the first three signed integers → (u, v, yaw). Unparseable output was treated as STOP in the reference evaluator.

coord_norm is true in config.json for this checkpoint (u/v are 0–1000); read it from the config rather than assuming.

Prompt format

The prompt is built through the checkpoint's own chat template (Qwen injects the default You are a helpful assistant. system turn, which matches training). The user turn is instruction first, then the ask, then the history frames, then the current frame:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
You are an autonomous navigation assistant.
{instruction}
Where should you go next to stay on track? Output the next waypoint's pixel coordinates (u v) in the current image and the relative facing direction at that waypoint (yaw, integer degrees, positive = right). Output STOP when finished.
These are your historical observations: <image>
<image>
... you can see <image>.<|im_end|>
<|im_start|>assistant

{instruction} is a compact-JSON prompt_body (whitelist keys in natural order, no spaces): {"goal":{"type":"<object|area>","value":"<caption>"},"route":[...],"constraints":[],"end_pose":[...],"quality":5}. Note the goal dict key is value, and always request quality: 5 at inference. route/end_pose may be empty (the gold-VLN eval drops them); an end_pose anchor [{"target_type":"object","target":"<name>","relation":"front","distance_m":<m>}] is understood for fine end-pose control (trained on objgoal data).
At step 0 the These are your historical observations: line is omitted — just {conjunction}<image>. after the ask.
The connector before the current frame (you can see ) is one of several training paraphrases; fix one canonical phrase for deterministic inference.

Observation preprocessing (evaluation parity)

setting	value
camera	egocentric RGB, 110° HFOV, single front view, camera height 0.88 m
sensor	720 (W) × 640 (H)
history	up to 8 frames, indices `np.unique(np.linspace(0, step_id-1, 8))` (uniform stride over the episode so far, excluding the current frame)
resize	history frames → 256×256, current frame → 512×512 (bilinear)
generation	greedy (`do_sample=False`), `max_new_tokens=32`, bf16
control	project the picked `(u, v)` + depth → world goal, drive there with a discrete-action controller, then rotate to `compass + yaw`

Reproducibility notes (benchmark-exact details)

Two details of the reference evaluator matter if you aim to reproduce the benchmark numbers exactly (rather than just deploy the model):

Connector phrase was randomized. The benchmark run drew the connector before the current frame per query from several training paraphrases ("you can see ", "in front of you is ", "there is ", "you can spot ", "you are toward the ", "ahead of you is "), not the fixed "you can see " shown above. Fixing one phrase is the right choice for deployment but is not bit-identical to the benchmark protocol.
Goal execution was capped. Each predicted waypoint was executed by unprojecting (u, v) against depth, snapping the world goal to the navmesh, then driving with a shortest-path follower capped at 10 discrete steps per goal, followed by a yaw-rotation phase capped at 12 turn steps; the model is then re-queried. Unbounded following (or skipping the navmesh snap) changes SR slightly.

Usage

python

import re
import torch
from PIL import Image
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor

REPO = "anchiehc/capra-v3-4b-n8"
model = Qwen3VLForConditionalGeneration.from_pretrained(
    REPO, torch_dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()
processor = AutoProcessor.from_pretrained(REPO)
coord_norm = bool(getattr(model.config, "coord_norm", True))
SENSOR_W, SENSOR_H = 720, 640

# --- observations: list of (frame_index, PIL.Image), oldest first
history = [(0, Image.open("frame000.jpg")), (4, Image.open("frame004.jpg"))]
current = Image.open("frame008.jpg")

instruction = ('{"goal":{"type":"object","value":"the grand piano in the middle '
               'of the room"},"route":[],"constraints":[],"end_pose":[],"quality":5}')
ask = ("Where should you go next to stay on track? Output the next waypoint's "
       "pixel coordinates (u v) in the current image and the relative facing "
       "direction at that waypoint (yaw, integer degrees, positive = right). "
       "Output STOP when finished.")

if history:
    hist_imgs = "".join("<image>\n" for _ in history)
    question = (f"You are an autonomous navigation assistant.\n{instruction}\n{ask}\n"
                f"These are your historical observations: {hist_imgs}. you can see <image>.")
    images = [im.resize((256, 256)) for _, im in history] + [current.resize((512, 512))]
else:
    question = (f"You are an autonomous navigation assistant.\n{instruction}\n{ask}\n"
                f"you can see <image>.")
    images = [current.resize((512, 512))]

parts = re.split(r"(<image>)", question)
content, k = [], 0
for p in parts:
    if p == "<image>":
        content.append({"type": "image", "image": images[k]}); k += 1
    elif p:
        content.append({"type": "text", "text": p})
text = processor.apply_chat_template([{"role": "user", "content": content}],
                                     tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=images, return_tensors="pt").to("cuda")
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=True)
decoded = processor.tokenizer.decode(
    out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

# --- parse (arrows -> STOP -> first 3 ints); yaw positive = LEFT
def parse(t):
    arrows = [c for c in t if c in "↑←→"]
    if arrows:
        return ("turn", arrows)
    if "STOP" in t.upper():
        return ("stop",)
    nums = list(map(int, re.findall(r"-?\d+", t)))
    if len(nums) < 3:
        return None
    u, v, yaw = nums[:3]
    if coord_norm:
        u = round(u / 1000 * SENSOR_W); v = round(v / 1000 * SENSOR_H)
    yaw = ((yaw + 180) % 360) - 180  # positive = LEFT
    return ("waypoint", u, v, yaw)

print(decoded, "->", parse(decoded))

Notes:

The predicted pixel is in the native sensor resolution (720×640), not the resized image handed to the model — resize is only to control the visual token count. Unproject (u, v) against depth at the native resolution.
In closed-loop use, re-query after executing (part of) the action, appending the new frame to the history.

Training

Base: Qwen/Qwen3-VL-4B-Instruct, finetuned (stock architecture, no vocab extension), bf16, 1 epoch.
Data: rendered VLN-CE-style corpora (R2R, RxR, ScaleVLN, VLNVerse) plus object-goal navigation, with the v3 pixel-goal text target ("u v yaw" / arrows / STOP); anchor-drop augmentation on the VLN rows (empty route/end_pose), force_quality=5, single 88 cm / 0° front rig.
Objective: next-token cross-entropy on the assistant text (teacher forcing).

Files

file	purpose
`model-*.safetensors`, `model.safetensors.index.json`	weights (bf16)
`config.json`	includes `coord_norm: true`
`tokenizer*`, `vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`, `chat_template.jinja`	text tokenizer
`preprocessor_config.json`	image processor
`generation_config.json`	generation defaults (overridden to greedy at inference)