You can deploy this model and start earning money today!

DangIT02/
qwen3vl-flowchart-to-mermaid_v3

Name: qwen3vl-flowchart-to-mermaid_v3
Availability: InStock

Views

No views yet

Qwen3-VL-8B — Flowchart to Mermaid (v3)

Vision-language model finetuned to transcribe flowchart diagram images into Mermaid flowchart code.

Given a flowchart image, the model produces valid Mermaid code that reproduces the diagram's nodes, edges, labels, and direction. This v3 release fine-tunes the vision tower (unlike v2 which kept it frozen) and substantially improves transcription fidelity, especially on simple diagrams.

Model details


Base model	`unsloth/Qwen3-VL-8B-Instruct`
Finetuning method	LoRA (rank 16, alpha 16, dropout 0.1), then merged to bf16
Trainable params during LoRA	51.3M / 8.82B (0.58%)
Layers finetuned	Vision tower + language layers (attention + MLP)
Dataset	`DangIT02/flowchart-to-mermaid_v2`
Split sizes	Train 4,511 / Val 655 / Test 662
Training	2 epochs, 564 steps, ~2h 25m on A100 80GB
Effective batch size	16 (per-device 4 × grad-accum 4)
Learning rate	3e-5, cosine schedule, 5% warmup
Final train loss	0.310
Final eval loss	0.221 (monotonic decrease, no overfit)
Peak VRAM	61.9 GB

Mermaid canonicalization

Important: This model outputs Mermaid code with canonicalized node IDs — A, B, C, ..., Z, AA, AB, ... in order of first appearance. Training data was canonicalized for tool-use compatibility and deterministic output.

Node labels (text inside shapes) and edge labels are preserved exactly. Rendering produces visually identical output to code with descriptive IDs.

If your downstream tooling requires semantic IDs, apply a post-processing rename based on labels.

Evaluation (full test set, n=662, canonicalized F1)

Metric	Overall	Small (<10)	Medium (10-20)	Large (20+)
node_f1	0.825	0.872	0.833	0.778
edge_f1	0.336	0.471	0.309	0.244
labeled_edge_f1	0.267	0.433	0.227	0.158
direction_match	0.899	0.990	0.986	0.744
parse_success	1.000	1.000	1.000	1.000

Sample counts per bucket: small 208, medium 212, large 242.

Comparison with v2 (vision-frozen baseline)

Metric (overall)	v2 (vision frozen)	v3 (vision unfrozen)	Δ
node_f1	0.677	0.825	+0.148
edge_f1	0.289	0.336	+0.047
labeled_edge_f1	0.220	0.267	+0.047

Biggest improvement — small diagrams. v2 hallucinated 3.2× more nodes than present on simple flowcharts (<10 nodes). v3 correctly reads the image without over-generating:

Small-diagram metric	v2	v3	Δ
node_f1	0.506	0.872	+0.366
edge_f1	0.257	0.471	+0.214
labeled_edge_f1	0.220	0.433	+0.213

Fine-tuning the vision tower (rather than relying on the frozen pretrained vision encoder) was the critical change.

Usage

With transformers

python

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
import torch

model_id = "DangIT02/qwen3vl-flowchart-to-mermaid_v3"
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("flowchart.png").convert("RGB")

messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Convert this flowchart to Mermaid code."},
]}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_tensors="pt", return_dict=True,
).to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=False,
        repetition_penalty=1.15,
    )

mermaid_code = processor.decode(
    out[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)
print(mermaid_code)

With vLLM (faster batched inference)

python

from vllm import LLM, SamplingParams
from PIL import Image

llm = LLM(
    model="DangIT02/qwen3vl-flowchart-to-mermaid_v3",
    dtype="bfloat16",
    max_model_len=16384,
    limit_mm_per_prompt={"image": 1},
)

sampling = SamplingParams(max_tokens=2048, temperature=0.0, repetition_penalty=1.15)

image = Image.open("flowchart.png").convert("RGB")
prompt = {
    "prompt": "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Convert this flowchart to Mermaid code.<|im_end|>\n<|im_start|>assistant\n",
    "multi_modal_data": {"image": image},
}
output = llm.generate([prompt], sampling)[0]
print(output.outputs[0].text)

Note on image size. For tall flowcharts (e.g. 1568×6376), start the vLLM server with a larger max-model-len (e.g. 32768) to avoid hitting the default 8192 token limit after vision encoding. Do not downscale aggressively — the model was trained at native resolution and loses fidelity on low-resolution inputs.

Recommended prompt templates

Any of these works — the model was trained on a variety of phrasings:

Convert this flowchart diagram to Mermaid code.
Generate the Mermaid code for the provided flowchart.
Analyze this flowchart and output the equivalent Mermaid code.
What is the Mermaid representation of this flowchart?
Transcribe this flowchart into Mermaid syntax.

Limitations

Direction detection degrades on large diagrams. For flowcharts with 20+ nodes, direction accuracy drops to 74.4% (compared to 99%+ on small/medium). The model sometimes defaults to graph TD when the true direction is BT or LR. If direction matters for your pipeline, include an explicit hint in the prompt or post-process.
Hallucination on very complex flowcharts. For diagrams with ≥25 nodes and dense text, the model occasionally generates plausible-but-incorrect flow structure (e.g., invents extra security steps, retry loops) instead of strictly transcribing. Affects <10% of large-bucket samples.
Canonicalized node IDs only. The model will not reproduce descriptive IDs from the original diagram. If your downstream tooling requires semantic IDs, post-process the output.
English labels only. Dataset is English-only; performance on diagrams with labels in other languages is untested.
Single-image input. Model accepts one flowchart image per prompt.
Max output ~2048 tokens. Very large flowcharts (>70 nodes) may be truncated.

Training recipe

Trained with Unsloth + TRL's SFTTrainer.

Key hyperparameters:

python

# LoRA
lora_rank=16, lora_alpha=16, lora_dropout=0.1
finetune_vision_layers=True        # key change vs v2
finetune_language_layers=True
finetune_attention_modules=True
finetune_mlp_modules=True

# Training
num_epochs=2
per_device_train_batch_size=4
gradient_accumulation_steps=4      # effective batch = 16
learning_rate=3e-5                 # lower than v2 to protect vision tower
weight_decay=0.01
warmup_ratio=0.05
lr_scheduler_type="cosine"
max_grad_norm=1.0
optim="adamw_8bit"

max_seq_length=8192
seed=3407

Ground-truth Mermaid code was canonicalized before training (node IDs → A, B, C, …) for deterministic output compatible with downstream tool use.

Why unfreeze vision?

v2 of this model froze the vision tower and achieved node_f1=0.677 on the test set. On downstream FlowVQA benchmarks it scored 72.95%, well below the v1 baseline (81.1%). Qualitative analysis showed the frozen vision encoder was insufficient for dense text-heavy flowcharts — the model compensated by pattern-matching text priors, producing plausible-but-wrong content.

v3 unfreezes the vision tower with a conservative learning rate (3e-5 vs 5e-5 in v2) and lower LoRA rank (16) to avoid degrading the pretrained vision weights. The result is +14.8 pp node_f1 overall and +36.6 pp on small diagrams.

Model versions

Version	Strategy	Test node_f1	Notes
v1	Unfreeze vision, 3 epochs, LR 1e-4	~0.70	Over-trained, eval_loss rose after step 500
v2	Freeze vision, 2 epochs, LR 5e-5	0.677	Hallucinated on simple diagrams
v3	Unfreeze vision, 2 epochs, LR 3e-5	0.825	Current recommended version

Citation

bibtex

@misc{qwen3vl_flowchart_mermaid_v3,
  title={Qwen3-VL-8B Flowchart-to-Mermaid (v3)},
  author={DangIT02},
  year={2026},
  howpublished={\url{https://huggingface.co/DangIT02/qwen3vl-flowchart-to-mermaid_v3}},
}

Base model: Qwen3-VL.