Qwen3vl Flowchart To Mermaid v5

Qwen3-VL-8B Flowchart → Mermaid (v5)

Fine-tuned Qwen3-VL-8B-Instruct for converting flowchart images into Mermaid code.

TL;DR

Metric	V5
node_f1	0.964
edge_f1	0.689
labeled_edge_f1	0.609
direction_match	1.000
parse_success	1.000

Quick start (vLLM)

vllm serve DangIT02/qwen3vl-flowchart-to-mermaid-v5 \
    --port 8000 \
    --max-model-len 8192 \
    --limit-mm-per-prompt image=1 \
    --gpu-memory-utilization 0.85

from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1", api_key="empty")

with open("flowchart.png", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="DangIT02/qwen3vl-flowchart-to-mermaid-v5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{img_b64}"}},
            {"type": "text", "text": "Convert this flowchart diagram to Mermaid code."},
        ],
    }],
    temperature=0.0,
    max_tokens=2048,
)
print(response.choices[0].message.content)

Quick start (transformers)

from transformers import AutoModelForImageTextToText, AutoProcessor
import torch
from PIL import Image

model_id = "DangIT02/qwen3vl-flowchart-to-mermaid-v5"
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("flowchart.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Convert this flowchart diagram to Mermaid code."},
    ],
}]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True).to("cuda")
output = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Model details

• Base model: unsloth/Qwen3-VL-8B-Instruct
• Task: Image-to-code (flowchart image → Mermaid syntax)
• Output direction: TD (top-down only)
• Output syntax: 100% pipe syntax A -->|label| B (no alternate A -- label --> B)
• Node ID style: Descriptive (Start, CheckInput) — not canonicalized

Training

Dataset

DangIT02/flowchart-to-mermaid_v2_2 (private, 2,872 samples):

• Train: 2,304 (587 easy / 906 medium / 811 hard)
• Val: 284 (70/113/101)
• Test: 284 (68/114/102)
• 100% TD direction, 100% pipe syntax

Hyperparameters

Param	Value
LoRA rank	32
LoRA alpha	32 (α=r)
LoRA dropout	0.05
Vision layers	unfrozen (trained)
Language layers	unfrozen (trained)
Learning rate	5e-5
LR scheduler	cosine
Warmup ratio	0.05
Weight decay	0.01
Max grad norm	0.3
Optimizer	adamw_8bit
Effective batch size	8 (2 × 4 grad_accum)
Epochs	2
Max sequence length	8192
Precision	bf16
Hardware	NVIDIA A100 80GB
Training time	98 min
Final train loss	0.286

Framework

• Unsloth for memory-efficient fine-tuning (2× faster, 60% less VRAM)
• TRL SFTTrainer with UnslothVisionDataCollator
• transformers 4.57.1, trl 0.22.2

Evaluation

Test set metrics (n=284, V2.2 clean test)

Metric	Mean	Std
node_f1	0.964	0.073
edge_f1	0.689	0.313
labeled_edge_f1	0.609	0.368
direction_match	1.000	0.000
parse_success	1.000	0.000

Per-complexity breakdown

Bucket	n	node_f1	edge_f1	labeled_edge_f1
small (<10 nodes)	91	0.976	0.897	0.863
medium (10-20 nodes)	98	0.974	0.781	0.710
large (20+ nodes)	95	0.941	0.396	0.262

Comparison with prior versions

Version	node_f1	edge_f1	labeled_edge_f1	Notes
V1	0.178	0.112	0.062	Baseline (V1.0 dataset)
V3	0.825	0.336	0.267	V2.0 dataset (BT-inflated test)
V4	0.826	0.451	0.382	V2.1 dataset (canonicalize bug)
V5	0.964	0.689	0.609	V2.2 clean dataset, no canonicalize

V5 improves edge_f1 by +53% and labeled_edge_f1 by +59% over V4 by:

1. Using V2.2 dataset (alt-syntax normalized to pipe syntax)
2. Disabling canonicalize_mermaid (model learns descriptive IDs from raw data)

Intended use

• Convert flowchart screenshots/diagrams to executable Mermaid code
• Build AI-assisted flowchart editing tools
• Document understanding for technical diagrams
• Pipeline component for diagram-to-text question answering

Limitations

1. TD direction only. Model trained exclusively on top-down flowcharts. Bottom-top, left-right, right-left flows may have lower accuracy.
2. Edge accuracy drops on large flowcharts (≥20 nodes): edge_f1 drops to 0.40 vs 0.90 for small. Use complexity-aware confidence thresholds in production.
3. Synthetic training data. All 2,872 training images are LLM-generated + Mermaid-rendered. Real-world handwritten or scanned flowcharts may show domain gap.
4. English labels only. Training labels are 100% English. Other languages not tested.
5. Mermaid output only. Does not support PlantUML, Graphviz, or other diagram DSLs.
6. No subgraph support. Training data has minimal subgraph usage; complex hierarchical diagrams may flatten.

Training pipeline

Qwen3-VL-8B base model
    ↓
LoRA r=32 α=32 (vision + language unfrozen)
    ↓
Train on V2.2 dataset (2,304 samples, 2 epochs, ~98 min on A100)
    ↓
Merged 16-bit checkpoint (~16 GB)
    ↓
This model

Output format

graph TD
    Start([Start]) --> CheckInput{Is input valid?}
    CheckInput -->|Yes| ProcessData[Process data]
    CheckInput -->|No| ShowError[Show error]
    ProcessData --> End([End])
    ShowError --> End

Format characteristics:

• graph TD direction (top-down)
• Descriptive node IDs (Start, CheckInput, not A, B)
• Pipe-style edge labels: -->|label|
• Standard shapes: [rect], {rhombus}, (rounded), ([stadium])

Citation

@misc{qwen3vl-flowchart-mermaid-v5,
  title={Qwen3-VL-8B Fine-tuned for Flowchart-to-Mermaid Generation},
  author={Nguyen Hai Dang},
  year={2026},
  howpublished={\url{https://huggingface.co/DangIT02/qwen3vl-flowchart-to-mermaid-v5}},
}

License

Apache 2.0 (inherited from Qwen3-VL base).

Acknowledgments

• Qwen team for Qwen3-VL-8B base model
• Unsloth for efficient fine-tuning framework
• Mermaid.js for the diagram syntax specification