You can deploy this model and start earning money today!

DangIT02/
qwen3vl-flowchart-to-mermaid_v2

Name: qwen3vl-flowchart-to-mermaid_v2
Availability: InStock

Views

No views yet

Qwen3-VL-8B — Flowchart to Mermaid (v2)

Vision-language model finetuned to transcribe flowchart diagram images into Mermaid flowchart code.

Given a flowchart image, the model produces valid Mermaid code that reproduces the diagram's nodes, edges, labels, and direction.

Model details


Base model	`unsloth/Qwen3-VL-8B-Instruct`
Finetuning method	LoRA (rank 16, alpha 16, dropout 0.1), then merged to bf16
Trainable params during LoRA	43.6M / 8.81B (0.50%)
Layers finetuned	Language (attention + MLP). Vision tower frozen.
Dataset	`DangIT02/flowchart-to-mermaid_v2`
Split sizes	Train 4,511 / Val 655 / Test 662
Training	2 epochs, 564 steps, ~3h 37m on A100 80GB
Effective batch size	16 (per-device 2 × grad-accum 8)
Learning rate	5e-5, cosine schedule, 5% warmup
Final train loss	0.070
Final eval loss	0.172 (monotonic decrease)
Peak VRAM	31.9 GB

Mermaid canonicalization

Important: This model outputs Mermaid code with canonicalized node IDs — A, B, C, ..., Z, AA, AB, ... in order of first appearance. This is because training data was canonicalized for tool-use compatibility and deterministic output.

Node labels (text inside shapes) and edge labels are preserved exactly.

If you need descriptive IDs (e.g. Start, ProcessPayment), apply a post-processing step to rename nodes based on their labels.

Evaluation (full test set, n=662, canonicalized F1)

Metric	Overall	Small (<10)	Medium (10-20)	Large (20+)
node_f1	0.677	0.506	0.796	0.719
edge_f1	0.289	0.257	0.359	0.256
labeled_edge_f1	0.220	0.220	0.284	0.164
direction_match	0.964	1.000	1.000	0.901
parse_success	1.000	1.000	1.000	1.000

Sample counts per bucket: small 208, medium 212, large 242.

Key observations:

Medium-complexity flowcharts (10-20 nodes) are the sweet spot — node F1 ≈ 0.80.
Large diagrams (20+ nodes) retain strong node accuracy (0.72) but direction detection drops to 90% — the model sometimes defaults to graph TD when the image uses BT or LR.
Small diagrams score lower because the model tends to over-generate (hallucinate additional nodes). See Limitations.
100% parse success — every generated output is valid Mermaid syntax.

Usage

With transformers

python

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
import torch

model_id = "DangIT02/qwen3vl-flowchart-to-mermaid_v2"
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("flowchart.png").convert("RGB")
if max(image.size) > 1024:
    image.thumbnail((1024, 1024), Image.LANCZOS)

messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "Convert this flowchart to Mermaid code."},
]}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_tensors="pt", return_dict=True,
).to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=2048,
        do_sample=False,
        repetition_penalty=1.15,
    )

mermaid_code = processor.decode(
    out[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)
print(mermaid_code)

With vLLM (faster batched inference)

python

from vllm import LLM, SamplingParams
from PIL import Image

llm = LLM(
    model="DangIT02/qwen3vl-flowchart-to-mermaid_v2",
    dtype="bfloat16",
    max_model_len=8192,
    limit_mm_per_prompt={"image": 1},
)

sampling = SamplingParams(max_tokens=2048, temperature=0.0, repetition_penalty=1.15)

image = Image.open("flowchart.png").convert("RGB")
prompt = {
    "prompt": "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Convert this flowchart to Mermaid code.<|im_end|>\n<|im_start|>assistant\n",
    "multi_modal_data": {"image": image},
}
output = llm.generate([prompt], sampling)[0]
print(output.outputs[0].text)

Recommended prompt templates

Any of these works — the model was trained on a variety of phrasings:

Convert this flowchart diagram to Mermaid code.
Generate the Mermaid code for the provided flowchart.
Analyze this flowchart and output the equivalent Mermaid code.
What is the Mermaid representation of this flowchart?
Transcribe this flowchart into Mermaid syntax.

Limitations

Hallucinates on very simple diagrams. On flowcharts with <10 nodes, the model tends to generate 2-3× more nodes than present (mean ratio 3.2×). Likely caused by training-data distribution skew toward medium/complex flowcharts (25.7% easy vs 74.3% medium/hard).
Direction detection degrades on large diagrams. For flowcharts with 20+ nodes, direction accuracy drops from 100% to 90%. The model sometimes defaults to graph TD when the true direction is BT or LR. Workaround: include the expected direction in the prompt.
Canonicalized node IDs only. The model will not reproduce descriptive IDs from the original diagram. If your downstream tooling requires semantic IDs, post-process the output.
English labels only. Dataset is English-only; performance on diagrams with labels in other languages is untested.
Single-image input. Model accepts one flowchart image per prompt.
Max output ~2048 tokens. Very large flowcharts (>70 nodes) may be truncated.

Training recipe

Trained with Unsloth + TRL's SFTTrainer.

Key hyperparameters:

python

lora_rank=16, lora_alpha=16, lora_dropout=0.1
finetune_vision_layers=False
finetune_language_layers=True
finetune_attention_modules=True
finetune_mlp_modules=True

num_epochs=2
per_device_train_batch_size=2
gradient_accumulation_steps=8    # effective batch = 16
learning_rate=5e-5
weight_decay=0.001
warmup_ratio=0.05
lr_scheduler_type="cosine"
max_grad_norm=1.0
optim="adamw_8bit"

max_seq_length=4096
seed=3407

Ground-truth Mermaid code was canonicalized before training (node IDs → A, B, C, …) for deterministic output compatible with downstream tool use.

Citation

If you use this model, please cite:

bibtex

@misc{qwen3vl_flowchart_mermaid_v2,
  title={Qwen3-VL-8B Flowchart-to-Mermaid (v2)},
  author={DangIT02},
  year={2026},
  howpublished={\url{https://huggingface.co/DangIT02/qwen3vl-flowchart-to-mermaid_v2}},
}

Base model: Qwen3-VL.