Qwen3-VL-8B — Constat Amiable Extractor (LoRA adapter)
A LoRA fine-tune of Qwen3-VL-8B-Instruct that reads a photo or scan of a
French constat amiable d'accident automobile (the European Accident
Statement) and returns a single JSON object with a fixed 118-field schema —
handwritten text, printed text, and checkbox states.
[!IMPORTANT]
This model was trained and evaluated entirely on synthetic data. No
real accident reports were used at any stage. The evaluation below measures
performance on the synthetic distribution the model was trained on — it
is not a measurement of real-world accuracy. Please read
Limitations & honest caveats before using or
citing these numbers.
200 optimizer steps (~0.23 epoch), 17 h on one NVIDIA DGX Spark (GB10)
Task
image → 118-field JSON (73 text fields + 45 checkboxes)
Released artifact
LoRA adapter only (~205 MB) — apply on top of the base model
Languages
French form; field content is mostly French, with some DE / EN / ES / IT / NL names
The point of this project is to show that a small, cheap, fully-synthetic
LoRA fine-tune can specialize an open 8B vision-language model on a narrow
document-extraction task. It is a methodology demonstration and a portfolio
artifact — not a claim of production-grade or real-world state-of-the-art
accuracy.
The constat amiable is the standardized European Accident Statement that two
drivers fill in by hand after a road accident. It is a dense, two-column form:
identity and address of each party, vehicle and insurance details, a 17-item
list of accident circumstances expressed as checkboxes per vehicle, free-text
observations, and a sketch ("croquis") of the scene.
Given one image of such a form, the model produces one JSON object:
text fields — transcribed verbatim (names, addresses, plate numbers,
dates, policy numbers, free-text observations…), or null when the field is
blank or illegible;
checkbox fields — true / false for every checkbox, including the 17
circumstance boxes for each vehicle.
The model is driven by a fixed system prompt (a "forensic document examiner"
role with anti-hallucination, checkbox and JSON-formatting protocols) plus a
short user instruction. Both are shipped in prompts/ and must be used at
inference time — the model was trained with them.
Intended use
Research and demonstration of synthetic-data specialization for VLMs.
Extracting fields from constat images of the template family seen in
training, in a setting where outputs are human-reviewed.
Out of scope
Production insurance processing without a prior validation campaign on
real, labelled constats.
Any automated legal, financial or claims decision.
Documents outside the trained template layouts, other form types, or other
languages.
Treating the JSON output as ground truth — VLM outputs must be verified.
Output schema
The schema has 118 keys in a flat object (no nesting). Breakdown:
The model was trained to emit raw JSON (no markdown fence, no preamble).
Greedy decoding (do_sample=False) is recommended for a deterministic,
reproducible extraction.
Training and the benchmark used the Unsloth 4-bit build
(unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit); 4-bit inference reproduces
the reported numbers and fits comfortably in <24 GB.
Training data
100 % synthetic, procedurally generated. No real constat was collected,
scanned or used. The generator (main.py) builds each sample from:
a small set of blank constat amiable form layouts — the standardized
European Accident Statement — used as the backgrounds onto which the
synthetic fields are rendered;
realistic fake content from Faker,
locale-weighted ~85 % fr_FR and ~3 % each of de_DE, en_GB,
es_ES, it_IT, nl_NL, plus domain dictionaries for vehicles, insurers,
European countries, postal-code formats, place names, etc.;
handwriting simulation — 304 fonts (filtered for French-glyph coverage)
to render the "handwritten" entries;
photo-realistic degradation applied with calibrated probabilities, so the
model sees the kind of images it would meet in the wild:
fold lines, coffee stains, rain drops, shadows & highlights, blur spots,
brightness/contrast variation, Gaussian blur, sensor noise, perspective
distortion, rotation (±6°), and JPEG recompression (quality 15–100, weighted
toward smartphone-grade 65–90);
final images resized to 1536–2240 px and aligned to the 32-pixel grid
Qwen3-VL requires.
Each sample ships with a perfect, machine-generated ground-truth JSON — the
key advantage of synthetic data: exhaustive, noise-free labels at scale, with
no privacy exposure (all identities are fake).
The held-out evaluation set (599 samples) was produced by a separate
generation run with the same generator — see Evaluation.
The honest trade-off: synthetic data is scalable and perfectly labelled, but
it only approximates real handwriting, real paper wear and real photo
conditions. The gap to real-world constats is real and is not measured
here.
Training procedure
QLoRA fine-tuning with UnslothFastVisionModel + TRL SFTTrainer. LoRA adapters were placed on both the
vision and the language layers (attention + MLP).
Loss. The first training steps logged a loss around 0.17; mean training loss
over the run was 0.0798, and validation loss (128 held-out samples) reached
0.0535 at the end of training.
Note: this is a deliberately light fine-tune — 200 steps is roughly a
quarter of one epoch. The validation-loss curve was still trending down, so
the figures below should be read as a lower bound on what this recipe can
reach with more training. Cross-entropy on a highly templated JSON target is
also a weak proxy for extraction quality — the real measure is the
field-accuracy benchmark below.
Evaluation
Methodology
The goal is an honest, transparent, auditable comparison. The benchmark
scripts are published in full (see
Benchmark scripts & transparency); the
synthetic evaluation set itself is not released.
Held-out set. A separate run of the same generator produced 599 fresh
synthetic constats (holdout_eval). Because every sample is an independent
random draw, these constats do not appear in the training set — they are a
held-out sample of the training distribution. The benchmark uses the
first 64 of them.
Models compared (same 64 images, same prompts):
Model
How it is run
Base — Qwen3-VL-8B-Instruct
zero-shot, greedy decoding, free-form JSON
Fine-tuned — this model
greedy decoding, free-form JSON (it was trained to emit the schema directly)
Gemini 3.1 Pro — gemini-3.1-pro-preview
Google AI API, structured output (response_schema), default sampling
Each model is evaluated in its realistic deployment mode. Gemini is given
structured output — a JSON schema it must conform to — because that is how
one would actually deploy it for extraction; this is the configuration that
gives Gemini its best, most reliable result (it is not a handicap). The
fine-tuned model uses free-form generation because it was trained to emit the
exact schema and does so reliably without constraint. All three receive the
identical system prompt and instruction.
Gemini settings.temperature and thinking_level left at their
defaults — Google explicitly advises against lowering the temperature for
Gemini 3, as it can cause looping/degradation. max_output_tokens = 32768. All
64 requests succeeded on the first call (0 retries, 0 API failures); the
retry logic in the script triggers only on transport/HTTP errors, never on
output quality.
Scoring. All three models are scored with the same metric definitions,
gathered in the module eval_metrics.py. Comparison is case-insensitive and
whitespace-normalized (a line break inside a form field is not a content
difference). With only 64 samples, the dominant source of uncertainty is
sampling noise, not the metric — see Limitations.
Metrics
Metric
Definition
json_valid
the output parses as JSON
field_acc
exact-match accuracy over all 118 fields
filled_acc
accuracy over only the fields that carry information in the ground truth (ticked checkboxes + non-empty text) — the honest "did it read the content" measure
text_acc
accuracy over the 73 text fields
checkbox_acc
accuracy over the 45 checkbox fields
cer
character error rate on text fields (Levenshtein ÷ length); lower is better
exact_full
1.0 only if all 118 fields are correct, else 0.0
Why two accuracy numbers? A real constat leaves many fields blank, and
"blank ↔ blank" counts as a correct match. So field_acc is optimistic —
it is partly rewarded for agreeing on empty fields. filled_acc is the
number that matters: accuracy on the content actually present on the form.
Results
Benchmark on n = 64 held-out synthetic constats.
Base vs fine-tuned vs Gemini 3.1 Pro — field extraction on the held-out synthetic set (n = 64)
Chart: CER is shown as character accuracy (1 − CER) so every bar points the
same way (higher = better); exact_full is omitted — it is ≈ 0 % for all three
models. Full numbers in the table.
Metric
Base Qwen3-VL-8B (zero-shot)
Fine-tuned (this model)
Gemini 3.1 Pro (structured output)
json_valid
93.75 %
100.00 %
100.00 %
field_acc (118 fields)
67.52 %
90.10 %
81.81 %
filled_acc (filled fields)
45.50 %
76.62 %
60.27 %
text_acc (73 text fields)
60.45 %
83.91 %
75.05 %
checkbox_acc (45 checkboxes)
78.44 %
99.63 %
92.23 %
cer (text, ↓ better)
0.3275
0.0970
0.1752
exact_full
0.00 %
1.56 %
0.00 %
exact_full is near zero for every model: it scores 1.0 only when all 118
fields are correct at once, so a single slip anywhere drops the sample to 0.
It is an intentionally brutal metric kept only for completeness — field_acc,
filled_acc and cer are the informative ones.
Interpretation
Fine-tuning delivers a large, consistent gain over the zero-shot base — on
every metric:
Metric
Base → Fine-tuned
field_acc
67.5 % → 90.1 % (+22.6 pts)
filled_acc
45.5 % → 76.6 % (+31.1 pts)
text_acc
60.5 % → 83.9 %
checkbox_acc
78.4 % → 99.6 %
cer
0.328 → 0.097 (text error rate down ~70 %)
json_valid
93.8 % → 100 %
This is the result the project set out to demonstrate: a light, fully-synthetic
LoRA fine-tune (200 steps) strongly specializes an open 8B vision-language model
for this extraction task.
On this benchmark the fine-tuned model matches or exceeds Gemini 3.1 Pro on
every metric (tied at 100 % JSON validity, ahead on all the others) — but that
comparison must be read carefully:
[!WARNING]
This is not a symmetric comparison. The fine-tuned model was trained on
this exact synthetic style and is evaluated on a held-out sample of the same
distribution. Gemini 3.1 Pro and the base model see this style for the first
time. The fine-tuned model's lead therefore measures how well fine-tuning
specialized it to the target distribution — it does not show that the
fine-tuned model is better than Gemini at reading real constats. Gemini's
score is best read as "what a strong general-purpose model achieves zero-shot
on this layout"; the fine-tuned score as "what task-specific specialization
achieves on its own distribution." A fair real-world comparison would need a
labelled set of real constats, which this project does not have (see
Limitations).
Honest takeaway. For a narrow, well-defined document-extraction task,
fine-tuning a small open model on synthetic data is remarkably effective: a
200-step LoRA was enough to take an 8B model from clearly behind to — on this
in-distribution benchmark — ahead of a frontier model, at a fraction of the
size and with no per-call API cost. How much of that advantage survives on
real-world documents is unmeasured, and is the main open question this project
does not answer.
filled_acc (76.6 %) is the most informative single number: of the fields that
actually carry content, roughly three in four are extracted exactly right —
strong for a 200-step fine-tune, with clear headroom remaining.
Limitations & honest caveats
Read this section before trusting any number above.
In-distribution evaluation — the comparison is not symmetric. The
fine-tuned model was trained on this exact synthetic style and is then
tested on a held-out sample of the same distribution. The base model and
Gemini 3.1 Pro see this style cold. A higher score for the fine-tuned model
therefore demonstrates successful specialization to the target
distribution — it does not establish that the fine-tuned model is
better than Gemini at reading real constats.
Synthetic ≠ real. Training and evaluation are 100 % synthetic. Real
constats have real handwriting, real paper wear, and real photo conditions
that the generator only approximates. Real-world accuracy is unmeasured.
No real labelled test set. The project has no corpus of real, annotated
constats, so no real-world claim can be made. Anyone deploying this model
should first run a validation campaign on real data.
Small evaluation set (n = 64). Point estimates carry a sampling
uncertainty on the order of several percentage points. Treat differences of
a few points as noise.
Light fine-tune. 200 steps ≈ 0.23 epoch; the model is under-trained
relative to the 14k samples available. The reported numbers are a lower
bound for this recipe, not its ceiling.
Template coverage. Only the ~5 template layouts seen in training are
supported. Other constat layouts, other insurers' forms, or other document
types are out of scope.
Hallucination. Like any VLM, the model can produce plausible-looking but
wrong field values, especially on illegible handwriting. The system prompt
instructs it to return null when unsure; this mitigates but does not
eliminate the risk. Outputs must be human-reviewed.
Privacy / GDPR. Training identities are fake (Faker). But real constats
contain personal data — names, addresses, plate numbers, dates of birth.
Any real-world use must comply with GDPR and applicable insurance-data
regulations.
License & legal
Adapter weights — released under Apache-2.0, inherited from the base
model Qwen/Qwen3-VL-8B-Instruct
(Apache-2.0). The adapter contains only low-rank weight deltas; it does not
embed any template image or any training sample.
Base model — you must obtain Qwen3-VL-8B-Instruct separately, under its
own Apache-2.0 license.
Datasets. The synthetic training and evaluation sets are not released.
This card describes how they were generated; the data itself is kept private.
Form. The constat amiable is the standardized European Accident
Statement — a public administrative form that insurers distribute freely to
every driver. The generator uses blank layouts of this standard form purely
as rendering backgrounds; the released adapter contains only weight deltas
and does not embed or reproduce any form image.
Benchmark scripts & transparency
All three benchmark scripts are published in full so the exact scoring
methodology can be read and audited:
File
Role
eval_metrics.py
the scoring metrics — shared module imported by both scripts below
eval_zeroshot_vs_finetuned.py
runs the base and fine-tuned models on the hold-out
eval_gemini.py
runs Gemini 3.1 Pro with structured output
Both scripts also write a per-sample *_details.jsonl — the raw model
prediction and the ground truth for every sample — used to inspect and verify
results.
[!NOTE]
The synthetic evaluation set itself is not released (see
License & legal). The scripts are published for
methodological transparency — so the scoring logic can be independently
checked — not as a turnkey reproduction package.
How the runs were launched:
bash
1# Base vs fine-tuned (same 64 hold-out images, greedy decoding)2EVAL_N=64EVAL_ADAPTER_PATH=./lora_model_XXXX \3 python eval_zeroshot_vs_finetuned.py
45# Gemini 3.1 Pro (same 64 images, structured output)6EVAL_N=64GEMINI_API_KEY=... \7 python eval_gemini.py
EVAL_N must be identical across the two scripts for the comparison to be
valid.
