A pipeline that restores damaged Indian temple imagery and projects the
result onto a 3D model. Rebuilt from the ARQGAN concept (2D restoration
GAN) for Kalinga/Nagara temple architecture, driven from an HBIM model.
Input: a temple render/photo with synthetic damage applied (cracks,
erosion, missing sections) + its segmentation map (per-architectural-
element mask, e.g. shikhara / mandapa / amalaka / plinth).
Output: the same view with damage removed, texture and fine
ornament detail restored by the trained HD generator.
Try it live: ArqGAN demo Space
— upload your own damaged temple photo and restore it in the browser.
Checkpoints in this repo
Temple
v2
HD
XL
Notes
checkpoints/ (Temple_1)
✅
✅
✅
fully trained baseline
checkpoints_Area_3_Temple_8/
✅
✅
✅
full v2/HD/XL run, fine-tune-cluster baseline (test SSIM 0.9902)
Only best_*.pt checkpoints are included (not last_*, which are
redundant duplicates from the same runs).
Dataset & demo
Dataset (3DITA source, point clouds, synthesized training triples) and
an interactive demo live separately:
3DITA point cloud + labels (per temple) 40-50 photos (per temple,
│ varying angle/lighting)
▼ │
dita_loader.py photo_alignment.py
Poisson mesh + per-face labels lighting normalize +
│ pose registration (PnP)
│ │
└──────────────────┬───────────────────────────────┘
▼
synthesize_pairs.py — thousands of (damaged, segmentation,
intact) training triples per temple
▼
train ArqGAN (v2 / HD / XL)
▼
apply to a damaged temple: mirror intact→broken (geometry),
GAN repaints rendered views, project back onto the mesh (texture)
▼
textured 3D model, viewable from any angle
Geometry comes from the point cloud, not the photos. 3DITA already
provides real, labeled geometry for all 47 temples — reconstructing it
again from photos would throw that away to rebuild a worse version.
Photos only ever contribute texture/appearance, after being told where
they were standing via classical pose registration.
What it is NOT
Not 3D generation. The GAN restores 2D images. 3D shape comes from
symmetry mirroring, not from any neural network inventing geometry.
Not trained on real damage. Training damage is simulated (synthetic
fractures + erosion). Real-world performance will differ — validate
against real damaged photos before trusting the numbers.
Not general. One temple's style. Kalinga-trained, won't transfer to
Dravidian/Hoysala/etc. without more data.
Folder map
Folder
What's in it
geometry/
Point cloud → mesh, photo → pose registration. Run this first if using 3DITA
core/
Base pipeline: data generation, the v2 model, training, HBIM loading, metrics, preflight check
hd/
The 18-hour coarse-to-fine model (512→1024, perceptual + feature-matching losses)
xl/
The bigger model (157M params, self-attention) + the ablation tool that checks whether bigger actually helped
to3d/
2D→3D texture projection: renders the mesh, runs the GAN, bakes colour onto the surface
docs/
Per-component READMEs, fix logs, and the overfitting/data guide
Root-level: smoke_test.py (post-install check), _paths.py (import
bootstrap — every script now runs from any working directory).
core/callbacks.py holds the shared training callbacks (early stopping,
checkpointing, the two overfitting detectors, LR schedule, logging).
Setup
bash
1pip install -r requirements.txt
2python smoke_test.py # 12 CPU checks, under a minute, no data needed
smoke_test.py verifies the plumbing — component transforms, the
validation split, all three model families, occlusion, the texture bake,
the pose round-trip. It says nothing about model quality; it catches the
failures that otherwise surface eighteen hours into a run.
Two optional-but-recommended extras:
embreex — accelerates occlusion testing. Occlusion is correct
without it, just slow; the code says which backend answered.
(An earlier version of this file recommended rtree here. That was
wrong — rtree is a spatial index, not a ray tracer, and trimesh's ray
casting never used it. See docs/05_BUGFIXES.md.)
xatlas — enables sharp UV texture maps (falls back to vertex colours
if absent OR if the build crashes; some builds segfault — the code
isolates it in a subprocess so a crash is survived, not fatal)
Checks GL context, file format, component names, and does a test render.
If it passes, everything downstream will work. Export your HBIM to GLB
first — IFC/RVT are not readable (preflight tells you this).
Step 0.5 — Build geometry + texture from 3DITA (if using point clouds)
Every epoch logs gen_gap (train minus val SSIM) and d_gap (D's score
on training reals minus held-out reals). Those two numbers are how you
tell memorisation from a hard task — the loss curves cannot. Training
ends with a pass on test wedges never used for early stopping or
checkpoint selection; that is the number to report, not best val.
See docs/06_OVERFITTING.md.
High-detail (HD, ~18h, better on fine ornament):
bash
1cd hd
2python train_hd.py --data_root ../data --total_hours 18
Use best_G.pt (v2) or best_local.pt (HD) for inference.
Step 3 — (optional) Did the bigger model help?
Train XL the same way (identical protocol, so the comparison is fair):
Renders the mesh, restores each view with the GAN, projects colour back
onto the surface. --lighting flat bakes clean albedo. Omit
--checkpoint for a passthrough test of the geometry path alone.
Hardware & time
Task
VRAM
Time
Data generation (3000 views)
CPU only
2–3 h
Train v2
~28 GB @ bs12
1.5–3 h (early-stops)
Train HD
~18 GB @ 1024/bs2
~14–16 h
Train XL
close to 32 GB
slower/epoch
2D→3D per temple
modest
20–30 min
Everything targets a single 16–32 GB GPU. Drop batch size if you OOM.
Estimated accuracy
Estimates, not measured — no training was run in the build environment.
Metric
Range
What it measures
Geometry whole-object F1@1%
90–96%
3D shape (from mirroring + hard-restore)
Geometry restored-region F1@1%
55–90%
the repaired region only; depends on damage symmetry
Texture SSIM (v2 / HD)
0.78–0.90 / 0.85–0.93
surface appearance from the GAN
Report whole-object AND restored-region side by side — the whole-object
figure is high mainly because measured geometry is preserved untouched.
Honest caveats (read before publishing any numbers)
All accuracy figures assume synthetic damage. Validate on real.
Geometry accuracy assumes symmetry holds — fails where both sides
are lost or the temple is intentionally asymmetric.
One-temple style bound. More reference temples = better coverage.
Nothing here was trained or benchmarked on a GPU in the build
environment. Verify the first epoch's timing and the 1024px forward
pass yourself before committing to a long run.
The 40–50 photos must be REAL photographs, not Blender renders.
They are the only place real-world surface appearance enters the
system; rendering them returns the mesh's own texture and the
pipeline becomes circular. Training triples, by contrast, are
correctly synthetic — paired damaged/intact ground truth cannot be
photographed. Full reasoning in docs/06_OVERFITTING.md.
Data generated before the correctness pass is invalid. Two bugs
silently corrupted it: component transforms were dropped (multi-part
HBIM exports collapsed to the origin) and segmentation maps were
rendered with shading (label colours varied by surface orientation).
Both are fixed; regenerate rather than reuse. Details in
docs/05_BUGFIXES.md.
See docs/ for detailed per-component notes and the fix logs.
docs/05_BUGFIXES.md is the one to read before running anything long.
