This model converts a photograph or scan of a handwritten UML class or activity diagram into compilable PlantUML code. It is a fine-tuned variant of Qwen3.5-4B, adapted in two stages:

Supervised fine-tuning (SFT) on ~500 handwritten UML diagrams paired with human-written PlantUML, teaching the base model the image-to-PlantUML task.
Group Relative Policy Optimisation (GRPO) with formalism-aware rewards: rather than scoring generated PlantUML as text, we compile it into an analysable representation (XMI for class diagrams, a custom-parsed control-flow graph for activity diagrams) and reward the model for structural and semantic agreement with a reference model, not surface similarity.

Full methodology, training details, and evaluation are described in the accompanying NIER paper.

Intended use

Converting handwritten UML class and activity diagram sketches (photos or scans) into PlantUML source
Research on formalism-aware or model-derived reward signals for structured code/model generation
Baseline or comparison point for further sketch-to-model fine-tuning work

Not intended for: diagram types outside class and activity diagrams (e.g. sequence, state machine, use case), typed/digital UML diagrams (the model was trained on handwritten sketches specifically), or general-purpose image-to-text tasks.

How to use

python

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "Seym0n/uml-to-plantuml-qwen3.5-4b"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {"role": "user", "content": [
        {"type": "image", "image": "path/or/url/to/sketch.jpg"},
        {"type": "text", "text": "Convert this handwritten UML diagram to PlantUML code."}
    ]}
]

inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True).to(model.device)
output = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Training data

Trained on CAS2UML, 557 hand-drawn UML diagrams (271 class, 286 activity) paired with manually validated PlantUML, released under CC-BY 4.0. Train/test split: 527 train, 30 held out (15 class, 15 activity).

Evaluation summary

Evaluated against the untuned Qwen3.5-4B base, GPT-4.1 Mini, and Gemini 3 Flash Preview on the 30-diagram held-out set, using both automatic formalism-aware metrics and a human ranking study (N=26). The fine-tuned model moves from last to second overall among the four, ahead of GPT-4.1 Mini and behind Gemini 3 Flash, with the largest gains in first-attempt compilation rate (class: 73.3% → 100%; activity: 20.0% → 86.7%). Full results, error analysis, and metric-validity discussion are in the paper.

Limitations

Evaluated on a small held-out set (30 diagrams); results are reported as emerging evidence, not a definitive benchmark.
The isolated contribution of the GRPO stage over SFT alone is not yet statistically distinguishable on this held-out set.
Errors cluster around fine visual detail (multiplicities, arrowhead types) rather than PlantUML syntax generation, pointing to a visual-grounding limitation rather than a language-generation one.
Trained and evaluated only on class and activity diagrams; not tested on other UML diagram types.

License

This model is released under Apache-2.0, matching its base model, Qwen3.5-4B. Note that the training dataset (CAS2UML) is released separately under CC-BY 4.0, and the PlantUML processing tooling used during training is released under GPL-3.0 — see the Zenodo archive for full artifact and license details.

Citation

bibtex

@inproceedings{sadeghi2026formalism,
  title     = {A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation},
  author    = {Sadeghi, Mersedeh and Scholz, Simon and Psoch-Bajraktari, Adrian},
  booktitle = {Proceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS 2026), NIER Track},
  year      = {2026}
}

Seym0n/uml-to-plantuml-qwen3.5-4b-gguf

From Sketch to Code: Fine-Tuned Qwen3.5-4B for Handwritten UML-to-PlantUML

Overview