Model lineage: this checkpoint was trained based on ZoomFly/rwkv-vl-1.5v100m-finevisionmax. It continues that model with the NV Image V3 reasoning-data stage; it is not a separately pretrained model.

The ZoomFly/rwkv-vl-1.5v100m-finevisionmax is better for

color-to-shape binding;
object appearance and local attributes;
preserving the earlier visual identity interface;
object-centric recognition tasks.

This model is better for

counting;
coarse horizontal/vertical localization;
comparing positions;
simple relation procedures;
visual arithmetic and multi-step chart reasoning.

Neither is sufficient for

general four-way coordinates;
systematic entity-to-region grounding;
robust multi-instance binding;
metric distance;
depth;
a compositional spatial coordinate algebra.

This is a research checkpoint of RWKV-VL 1.5B-v100M continued on vision-language data containing explicit <think>...</think> reasoning traces. It supports two inference modes from the same weights:

Thinking mode is the default. The model may produce a filled <think>...</think> trace before its final answer.
Direct mode is forced by prefilling an empty <think>\n</think>\n block. The model then answers without generating a reasoning trace.

The checkpoint demonstrates real gains from visual reasoning, especially on chart arithmetic, numerical calculation, and code reasoning. It does not reliably decide how much reasoning a question needs. Thinking can substantially hurt recognition, strict yes/no tasks, translation verification, and multi-image questions. Long traces can drift away from the requested task, repeat, or fail to close.

This model should therefore be used with caller-controlled mode routing. Direct mode is the safer default; enable thinking when a question genuinely requires several visual or computational steps.

At a Glance

Property	Value
Architecture	Frozen Qwen-style vision encoder + visual MLP projector + RWKV7 language model
Parameters	1,653,177,600
Weight dtype	BF16
Checkpoint size	Approximately 3.1 GB
RWKV hidden size / layers	2,048 / 24
Context length	8,192 tokens
Vocabulary size	65,536
Image budget	65,536 to 3,145,728 pixels per sample
Vision patch / merge size	16 / 2
EOS token IDs	10060 (`✿`) and 0 (`<
Default behavior	Generate `<think>...</think>`, then answer
Direct-mode prefix	`<think>\n</think>\n`

Which Mode Should I Use?

Task type	Recommended mode	Observed behavior
Chart questions requiring lookup plus arithmetic	Thinking	ChartQA rises from 52.0% direct to 66.8% thinking on a 250-example evaluation
Numerical calculation	Thinking	MME accuracy rises from 52.5% to 85.0%
Code reasoning	Thinking	MME accuracy rises from 40.0% to 75.0%
Counting or position questions that require comparison	Usually thinking, with a strict token cap	MME count rises by 10 points and position by 15 points
Straight OCR	Either; thinking gain is small	75.0% thinking versus 72.5% direct on 40 MME examples
Commonsense yes/no	Direct for efficiency	No measured accuracy gain from thinking
Object or scene existence	Direct	Thinking adds work and slightly reduces accuracy
Landmark, artwork, celebrity, scene, or poster recognition	Direct	Thinking causes hallucinated identification, task drift, and occasional repetition loops
Translation verification framed as yes/no	Direct	Thinking often performs a translation instead of answering yes/no
Simple visual lookup with a strict output format	Direct	Long reasoning may alter a correct first read or answer in the wrong format
Multi-image comparison, temporal frames, maps, or grounding	Direct, but treat this checkpoint as experimental	On the balanced multi-image diagnostic, direct scores 29.5% and thinking 10.5%

These recommendations are empirical routing rules for this checkpoint, not a claim that reasoning is inherently harmful for recognition or multi-image tasks.

Quick Start

The repository contains custom model and processor code. Review the Python files and use trust_remote_code=True only when you trust the repository.

The following versions were used to validate this export:

bash

pip install "transformers==5.14.1" "accelerate==1.14.0" \
  "flash-linear-attention==0.5.0" pillow

Install a PyTorch build appropriate for your CUDA environment separately.

Thinking and Direct Modes

python

from pathlib import Path

import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor


model_id = "ZoomFly/rwkv-vl-1.5v100m-nv_img_v3-260815"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()


def ask(
    images: Image.Image | list[Image.Image],
    question: str,
    *,
    thinking: bool,
    max_new_tokens: int | None = None,
) -> str:
    if isinstance(images, Image.Image):
        images = [images]
    images = [image.convert("RGB") for image in images]

    content = [{"type": "image"} for _ in images]
    content.append({"type": "text", "text": question})
    messages = [{"role": "user", "content": content}]

    prompt = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    if not thinking:
        prompt += "<think>\n</think>\n"

    inputs = processor(
        text=[prompt],
        images=images,
        padding=False,
        return_tensors="pt",
    ).to(model.device)

    if max_new_tokens is None:
        max_new_tokens = 512 if thinking else 128

    with torch.inference_mode():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            use_cache=True,
        )

    generated_ids = output_ids[:, inputs.input_ids.shape[1] :]
    return processor.batch_decode(
        generated_ids,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )[0]


image = Image.open(Path("example.jpg"))

# Use thinking for questions that require several visual/computational steps.
print(
    ask(
        image,
        "Read the chart values, compute their median, and give the result.",
        thinking=True,
        max_new_tokens=768,
    )
)

# Use direct mode for recognition, lookup, and strict-format questions.
print(
    ask(
        image,
        "Is there a red car in the image? Answer only Yes or No.",
        thinking=False,
    )
)

The direct-mode prefix is part of the input, so the decoded continuation normally contains only the answer. Direct mode avoids generating a trace, but it does not restore the weights of the model before thinking training.

Multi-Image Input

Pass all images in order and include one image item per image in the message:

python

images = [
    Image.open("frame_1.jpg"),
    Image.open("frame_2.jpg"),
    Image.open("frame_3.jpg"),
]

answer = ask(
    images,
    "Which frame contains the white cabinet? Answer with 1, 2, or 3.",
    thinking=False,
)
print(answer)

The processor correctly preserves the number and order of image feature blocks. However, this checkpoint may skip an image, merge evidence across frames, or copy one description to several images during joint reasoning.

Evaluation Summary

Evaluations used lmms-eval commit v0.6-167-g6619ef61, greedy decoding, and the checkpoint's custom batch-invariant generation path. Reasoning tags were removed before task scoring.

Two modes were evaluated:

Thinking: normal generation prompt, up to 2,048 tokens for ChartQA and MME, or 1,024 tokens for the multi-image diagnostic.
Direct: generation prompt followed by <think>\n</think>\n, with up to 512 tokens. This was evaluated through a companion export with the prefix embedded in its chat template; the weights were identical.

ChartQA

This is a deterministic evaluation of the first 250 ChartQA test examples, not the complete benchmark.

Model	Thinking	Direct	Thinking minus direct
This checkpoint	66.8%	52.0%	+14.8 points
FineVisionMax comparison checkpoint	55.2%	53.2%	+2.0 points

For this checkpoint:

113/250 examples are correct in both modes.
54/250 are correct only with thinking.
17/250 are correct only in direct mode.
66/250 are wrong in both modes.
Thinking has a median output length of 237 tokens.
Direct mode has a median output length of 2 tokens.
6/250 thinking generations never close </think> and hit the 2,048-token limit.

Thinking helps when the model must read several values and then calculate or compare them. Thinking hurts when it selectively drops one of the values it already identified, misreads a chart value during a long trace, makes an arithmetic error, or reinterprets the question.

MME Mode Comparison

The following table reports per-question accuracy over all 2,374 MME questions. It is useful for comparing modes, but it is different from MME's official category-aggregated point score.

MME subtask	Samples	Thinking	Direct	Difference
Text translation verification	40	22.50%	72.50%	-50.00 points
Landmark recognition	400	66.25%	83.50%	-17.25 points
Artwork recognition	400	58.75%	75.00%	-16.25 points
Celebrity recognition	340	62.35%	75.59%	-13.24 points
Scene recognition	400	80.00%	85.75%	-5.75 points
Poster recognition	294	63.95%	68.71%	-4.76 points
Existence	60	93.33%	96.67%	-3.34 points
Commonsense reasoning	140	70.71%	70.71%	0.00 points
OCR	40	75.00%	72.50%	+2.50 points
Color	60	85.00%	76.67%	+8.33 points
Count	60	80.00%	70.00%	+10.00 points
Position	60	85.00%	70.00%	+15.00 points
Numerical calculation	40	85.00%	52.50%	+32.50 points
Code reasoning	40	75.00%	40.00%	+35.00 points
Concrete group	2,074	68.76%	78.30%	-9.54 points
Abstract group	300	67.33%	64.67%	+2.66 points
All questions	2,374	68.58%	76.58%	-8.00 points

The official MME point totals tell a complementary story because they weight and aggregate task categories differently:

Mode	Perception points	Cognition points	Total points
Thinking	1,317.67	441.07	1,758.74
Direct	1,361.77	351.43	1,713.20

Thinking improves the official cognition component enough to raise the total point score, while direct mode answers more individual questions correctly. Report the metric definition whenever citing either result.

Multi-Image Diagnostic

This is a paired, deterministic 200-example subset of the first official MuirBench parquet shard: 40 examples from each of five tasks, 2-9 images per example, seed 42. It is not the official full MuirBench score.

Task	Direct	Thinking
Action Understanding	32.5%	10.0%
Counting	25.0%	10.0%
Geographic Understanding	22.5%	2.5%
Image-Text Matching	42.5%	20.0%
Visual Grounding	25.0%	10.0%
Overall	29.5%	10.5%

The FineVisionMax comparison checkpoint reaches 37.0% semantic accuracy on the same subset. Its strict task score is only 3.5% because it often begins with The correct answer is B... instead of emitting a bare option letter; the 37.0% comparison extracts explicit answer statements while leaving unextractable responses wrong.

In thinking mode, only 79/200 traces close within 1,024 tokens. The remaining 121 hit the token limit without emitting a final answer. Of the 79 completed traces, 21 are correct. Increasing the reasoning budget increases the number of completed thoughts, but does not recover direct-mode accuracy.

All tested image blocks are present and causally affect recurrent state. The failure is not caused by the processor dropping or reordering images. The model instead has difficulty retaining distinct image identities and binding visual evidence to the requested answer during a long recurrent trace.

What Thinking Can Do Better

Thinking is most useful when a direct answer would otherwise be a guess and the trace can supply a checkable intermediate computation:

read multiple chart values, compare them, and calculate a result;
solve visual arithmetic or numerical verification questions;
inspect short code and reason about its output;
count, compare positions, or combine several observations;
explain an answer when a user benefits from intermediate work rather than only the final result.

Productive traces explicitly identify the relevant evidence, perform a short calculation or comparison, close </think>, and then commit to one answer.

Where Thinking Is Unreliable

Thinking is usually unnecessary or actively harmful when the answer depends on one stable visual judgment:

landmark, artwork, celebrity, poster, and scene recognition;
object existence and simple visual lookup;
strict yes/no questions;
translation-verification questions where the model may start translating instead of judging the proposed translation;
tasks requiring an exact short output format;
current multi-image comparison, temporal-frame, map, and grounding tasks.

Observed failure patterns include:

Task drift: the trace gradually answers a different question.
Selective evidence: the trace lists all relevant values but compares only a subset.
Accumulated visual error: a later reasoning step replaces a correct first read with a hallucinated value or identity.
Reasoning error: the trace performs an incorrect median, ratio, or arithmetic operation and then confidently follows it.
Image identity mixing: evidence from one image is copied to another or several frames receive the same description.
Runaway reconsideration: repeated Wait, Let's re-evaluate, or literal phrase/token loops prevent </think> closure.
Format mismatch: the semantic answer may be reasonable but omit a required unit, month, option letter, or yes/no form.

Known Structural Limitations

Controlled component swaps found that this checkpoint still recognizes simple visual attributes, but may fail to bind them to an answer choice:

text-only A-D mapping: 16/16;
single-image color naming: 4/4;
single-image visual color-to-A-D mapping: 4/16, versus 16/16 for the FineVisionMax comparison checkpoint.

The regression is reproduced by swapping only the trained RWKV recurrent stack into the comparison model; swapping only the projector or LM head does not reproduce it. This means direct mode avoids the generated trace but does not remove all effects of thinking training on the recurrent model.

Generation also showed residual batch-composition sensitivity in some greedy replays. The reported lmms-eval runs used the custom batch_invariant=true path, but some stopping outcomes still changed when batch companions or device ranks changed. For critical use, generate one sample at a time and verify the answer independently.

Flash Linear Attention warns that its RWKV implementation may differ from the official RWKV implementation. Results in this card use one consistent runtime for all paired comparisons, but absolute behavior may change with another kernel or implementation.

RWKV-VL does not expose a Transformer-style text-to-image attention matrix, and output_attentions=True is unsupported. Visual information is inserted as projected image-token embeddings and propagated through recurrent state.

Practical Generation Guidance

Route simple recognition and strict-format questions to direct mode.
Begin thinking mode with a 256-512 token budget; raise it only when the task demonstrably needs more steps.
Treat failure to close </think> as a failed generation instead of assuming that a larger budget will eventually produce a correct answer.
Prefer greedy decoding for reproducibility. Sampling may amplify drift and repetition in an already unstable trace.
Ask for one final answer after the reasoning trace and validate required formats in application code.
For multiple-choice tasks, parse explicit forms such as The correct answer is B instead of accepting only a leading letter.
For multi-image tasks, number images in the prompt, request one short evidence statement per relevant image, and still verify that descriptions are not duplicated across images.
Do not use chain-of-thought text as a factual explanation without checking it against the image and final answer.

Training Provenance

The available training log records:

initialization based on ZoomFly/rwkv-vl-1.5v100m-finevisionmax and continued with the NV Image V3 reasoning-data stage;
initialization from an earlier RWKV-VL training checkpoint using model weights only, with fresh optimizer and LR schedule state;
Nemotron-Image-Training-v3-parquet as the image-text source;
no text-only data mixture;
1,705 optimizer steps with global batch size 512;
sequence length 8,192;
frozen vision encoder;
projector LR 3e-5;
full RWKV/LLM LR 3e-5;
1,552,584,704 trainable parameters;
250 warmup steps.

The run processed approximately 872,960 examples before the data source was exhausted. The public source mixture is dominated by single-image examples; its sampled multi-image portion is primarily multi-page document data rather than independent-scene comparison or temporal-frame reasoning. This may not exactly match the post-filtered training snapshot.

Intended Use

Appropriate uses include:

research on recurrent vision-language reasoning;
comparing trace-generating and direct multimodal inference;
studying when visual chain-of-thought helps or hurts;
controlled experiments on reasoning length, stopping, and multi-image state;
non-critical chart, arithmetic, OCR, and visual question-answering prototypes with answer verification.

This checkpoint is not recommended for autonomous high-stakes decisions, unverified factual identification, safety-critical visual inspection, or applications that require stable hidden reasoning or guaranteed multi-image coverage.

Limitations and Safety

The model can hallucinate objects, values, identities, text, and actions.
It can emit persuasive but incorrect intermediate reasoning.
It may fail to stop or may repeat text until the generation limit.
Recognition of people, landmarks, artwork, and scenes is not reliable.
OCR, counting, chart reading, arithmetic, and option selection can be wrong.
Multi-image input is accepted, but all images may not be considered distinctly in the answer.
Outputs may reflect biases or unsafe content present in the pretraining and fine-tuning data.
The checkpoint has not been evaluated for fairness, privacy leakage, memorization, adversarial robustness, or comprehensive safety behavior.

License and Attribution

No license is declared in this model card. Before public release, the publisher should verify and add the licenses and attribution requirements for the base model, RWKV/FLA components, vision encoder, training data, and bundled custom code.