One Qwen2.5-VL-7B LoRA. One generation per plan. Metadata and geometry in the same JSON
object, from the page pixels alone.
bash
1python verify.py # 6 offline checks, ~2 min, no GPU — run this first2python make_views.py # cache 3 views per plan (877 plans)3python build_dataset.py # data/{train,val,test}.jsonl + folds.csv4python evaluate.py --oracle # end-to-end plumbing check: 0.99955python baseline_submission.py # the floor: 0.54826# then train_qwen_lora.ipynb on Colab/Kaggle, then:7python infer.py --adapter outputs/lora_final --split test.jsonl
8python postprocess.py --raw outputs/raw --out outputs/submission.csv
Read this before adding a segmentation model
Score = 0.5·IoU + 0.3·MCA + 0.2·(1 − WER/2)
Reverse-engineered to 9 decimal places from both reported leaderboards. metric.py reproduces
it exactly.
The IoU term compares masks from polygon_to_mask()
(utils/seg_utils.py:96),
which normalises every polygon into its own bounding box at 256×256 before comparing.
That single line is the whole design. Position, scale and aspect ratio are all invisible to
the score — verified: the axis-aligned bbox of the truth scores 0.554, identical to submitting
a constant unit square for every row. Only shape and rotation matter, and rotation is safe
because these plans are drawn grid-north-up (checked visually; both 7703-079 and 7704-115
print a "GRID NORTH" arrow pointing up, which is also why the reference solution could submit
raw pixel polygons and score 0.977 IoU).
So there is nothing to georeference, and therefore nothing to annotate or segment. The
geometry labels already exist in Train.csv; the target is that WKT shape, normalised.
Not built, deliberately, because the metric discards all of it: the 14-hour geo→pixel
alignment pass, the Unet++/EfficientNet-B5 segmentation model, YOLO labels, coordinate-box
OCR, the affine transform, the surveyor bias model, 8-way TTA. The existing
survey_plans_yolo_labels/ colour masks measure 0.555 best-instance shape-IoU against the
truth — no better than a constant rectangle, i.e. worthless.
Where the score actually is
term
weight
how it is earned
IoU
0.50
the model's shape, arbitrated between two heads
MCA
0.30
Parish + Unit of Measurement are free constants (0.12); only Certified date, Total Area, LT Num must be read
WER
0.20
but an empty string already scores 0.5 of it, so the real swing is 0.10
Parish is St. Philip for 662/662 train rows and Unit of Measurement is sq m for
661/662. postprocess.py forces both.
Measured floors and ceilings over the 662 unique train IDs:
prediction
mean IoU
total score
no model, unit square
0.575
0.5215
no model, mean train shape
0.628
0.5482 ← the floor to beat
min-rotated-rect of the truth
0.905
—
convex hull of the truth
0.981
—
truth + 2% vertex jitter
0.939
—
truth + 5% vertex jitter
0.862
—
this pipeline on ground-truth targets
0.999
0.9995 ← the ceiling
Median parcel is 8 vertices, 71% have ≤10. Vertices accurate to ~2% of parcel size give
~0.94 IoU, so the target is well within reach of an OCR-grade read.
Two geometry heads, arbitrated by closure
The model emits geometry twice, in the same JSON object:
polygon — vertices on a 0–1000 grid, aspect-preserving, clockwise from the
northernmost. Round-trips at 0.9979 IoU.
traverse — the bearings and distances printed on the plan, [["091°47'", 31.48], …].
Round-trips at 0.9991.
The traverse is there because it is self-checking. Ground-truth bearings are exactly whole
arcminutes (100% within 0.5′, mean residual 11.6″) and distances exactly 2 d.p., because the
truth was computed from that printed traverse in the first place. Walk the legs and a correct
read closes on itself; a misread bearing or a dropped leg leaves a gap. That gap
(closure_error = misclosure / perimeter) is a free confidence signal no other head has.
traverse_to_poly() distributes any residual misclosure over the legs in proportion to their
length — the compass (Bowditch) rule from ordinary survey practice. That is what makes the
head survive a dropped short leg: 16.7% of ground-truth edges are under 2 m, small enough
that the model will sometimes miss one.
Validated in verify.py step 4 against three simulated OCR failure modes — a dropped short
leg, a bearing off by 10°, a distance decimal slip:
n
mean IoU
closure < 0.02
907
0.9832
closure ≥ 0.02
719
0.8525
and a clean traverse never exceeds 3.7e-04, so τ=0.02 never rejects a good read. Bowditch is
worth 0.9668 vs 0.9515 unadjusted on the dropped-leg case.
The arbitration ladder (postprocess.py)
traverse, if closure < τ (0.02) and ≥4 legs
vertices from the polygon head
traverse anyway, even badly closed — still 0.853, well above the floor
convex hull of whatever raw points survived. Hull, not minimum_rotated_rectangle:
hull of the truth is 0.981 against the rect's 0.905, and on the thin L-shaped parcels the
rect actually scores 0.35–0.58, below the constant floor.
mean train shape — 0.628, and never a zero.
Every rung must clear a degeneracy test (fills ≥2% of its own rotated bbox; the thinnest real
parcel fills 33%), so a ring that repair flattened into a sliver falls through instead of
scoring near zero. An empty geometry is never emitted —
pipeline/output/results.csv currently has a POLYGON Z EMPTY row, which is a hard zero.
The duplicate-ID trap
74 of the 662 train IDs (39.1% of rows) carry several unrelated parcel geometries —
7710-064 has 39 — with sibling shapes agreeing at only 0.656 IoU and area ratios up to
4049×. The reference solution's blind drop_duplicates(subset=["ID"])
(create_dataset.py:42)
trains geometry on a coin flip there.
build_dataset.py instead identifies the subject parcel by matching polygon area to the
plan's own printed Total Area, which is in the labels. Honest validation, by planting
distractors on IDs that are not ambiguous:
candidates
picked correctly
coin flip
2
94.2%
50%
3
92.0%
33%
5
90.8%
20%
10
87.0%
10%
Applied conservatively (best within 10%, runner-up at least 2× further off) it resolves 28
of the 74. The remaining 46 become metadata-task samples — a prompt that never mentions
geometry — so their metadata still trains without teaching a bad shape. Text is
byte-identical across duplicate rows (verified), so only geometry was ever affected.
Note that evaluate.py reports the unambiguous subset separately, because
metric.truth_frame() keeps the first row per ID while we pick the area-matched one; on
those 28 the local truth is itself a coin flip (they score 0.705 against it). Test IDs are all
single-target, so this costs nothing on the leaderboard.
What the prompt gets right that pipeline/prompts.py does not
LT Num is not a guess. It is printed on every plan as Land Tax Ref: / Val. Ref # /
Tax Ref#, with - or / separators the labels write as dots. Verified on 7703-079
(Val. Ref # 77/03/09/068 → 77.03.09.068), 7707-115 and 7703-078. File No: on the
same panel is a decoy. It is 100% NN.NN.NN.NNN.
Certified date has decoys.7703-079 also prints "certified 10th April, 1975" inside
a Reference-plans note; 7707-115's Note cites a 1972 certification and its RECORDED stamp
reads a different day. The prompt asks for the date on the certified line above the
surveyor's signature.
Total Area is the Total line, not Land — plans split area into
Land / Verge / Carriageway.
Rule 10 of pipeline/prompts.py asks for real-world Eastings and Northings. Replaced: the
score cannot see position or scale.
prompts.py is imported by both the notebook and infer.py. Neither is allowed its own copy
— a prompt that drifts between training and inference silently asks the model a question it
was never taught.
Views and the token budget
Pages are 2480×3507 (A4 at 300 dpi) with the diagram at the top and the certification block
at the bottom, so one downscaled full page makes the small text unreadable. Three crops,
aspect-preserving and snapped to Qwen's 28 px token cell (14 px ViT patch, 2×2 merged):
view
crop
size
image tokens
full
whole page
728×1036
962
top
upper 60%, the diagram and its edge labels
1288×1092
1794
bottom
lower 45%, title and certification
1148×728
1066
3822
Hence max_seq_length: 6144, not the plan's original 4096. Letterboxing into squares would
be score-neutral (the metric is aspect-blind) but would spend ~30% of those tokens on white.
The cached views land in views/<ID>/{full,top,bottom}.jpg, 411 MB for all 877 plans. That
plus data/*.jsonl and the .py files is everything the GPU box needs — the original page
scans stay local.
4 train IDs have no page image (5608-085, 8608-048, 8616-30, 8616-31) and are dropped;
all 219 test IDs have one.
One deliberate non-repair
postprocess.py does not clamp out-of-range Certified date values to the observed
2013-02-04 … 2025-05-27 window, which the plan originally called for. A date outside it is
almost always a decoy read from a Notes paragraph or a RECORDED stamp, and clamping it to a
boundary is just as wrong as leaving it — MCA is exact-match, so both score zero. Clamping
would only hide the diagnostic. The count is reported instead.
