Fine-Grained Image Classification of World Architecture: An EfficientNetV2-S Transfer Learning Approach with Layered Regularization

Architectural Building Image Classifier

Model Description

• GeM Pooling (Radenovic et al., CVPR 2018) — replaces global average pooling with a learnable power parameter (p=3.0) that emphasizes high-activation features, yielding stronger discriminative representations for FGIC tasks
• Focal Loss (Lin et al., ICCV 2017, gamma=2.0) — down-weights well-classified examples to focus gradient updates on hard-to-classify building pairs
• DiscriminativeAdamW LR — extends AdamW with per-variable LR scaling on block6 (×0.1) via (update_step) override, combined with selective fine-tuning (block6+top_conv unfrozen, BN frozen). LR scaling produces truly discriminative updates — block6 variables receive 10× smaller learning rate than head variables (117 total: 105 block6 + 12 head)
• Mixup + CutMix (Zhang et al., ICLR 2018. Yun et al., ICCV 2019) — alternating per-batch (50/50): Mixup (alpha=0.2, linear interpolation) and CutMix (alpha=1.0, spatial patch). Applied only in Phase 1 training to regularize head learning
• Selective Unfreeze (Yosinski et al., 2014) — Phase 2 unfreezes block6+top_conv layers (180/513 EfficientNetV2-S layers) while keeping BatchNormalization frozen to preserve pretrained statistics
• SWA with BN re-estimation (Izmailov et al., UAI 2018) — 10-epoch post-training weight averaging with constant LR 1e-4, followed by 100-step batch normalization statistics re-estimation (3,200 images)
• Test-Time Augmentation — 6 variations averaged at inference: original, horizontal flip, center crop 85%, corner crop 70%, corner crop top-left 80%, corner crop bottom-right 80%. Yields +0.22% accuracy improvement (97.92% → 98.14%)
• Grad-CAM (Selvaraju et al., ICCV 2017) — gradient-weighted class activation mapping for explainability, targeting top_conv (last Conv2D layer of EfficientNetV2-S)
• ECE Calibration (Guo et al., ICML 2017) — Expected Calibration Error with 15-bin reliability diagram to assess prediction confidence reliability
• Temperature Scaling (Guo et al., ICML 2017) — post-hoc calibration via scalar temperature parameter T optimized on validation set (NLL minimization). T=0.4645 reduces ECE from 18.13% (underconfident due to Label Smoothing) to 0.95% — applied at inference via (softmax(log(probs) / T)) trick

Architecture

Input (320, 320, 3)
  │
  EfficientNetV2-S (ImageNet pretrained, 513 layers, 20.33M params)
  │
  Conv2D(256, 3×3, ReLU, padding=same)     →  2,949,376 params
  BatchNormalization                        →  1,024 params
  MaxPooling2D(2×2)                         →  0 params
  │
  GeM Pooling(p=3.0, eps=1e-6, learnable)  →  1 param
  │
  Dense(256, ReLU)                          →  65,792 params
  BatchNormalization                        →  1,024 params
  Dropout(0.4)                              →  0 params
  │
  Dense(8, Softmax)                         →  2,056 params
  │
Output (8 classes)

Performance

Overall Metrics

Per-Class Results

Model Selection

Training Progression

Phase 1 ran 25 epochs (maximal), best epoch = 23 with val_accuracy 97.54%. EarlyStopping with patience=5 was not triggered. Phase 2 ran 7 epochs, best epoch = 2 (val_accuracy 97.77%), EarlyStopping with patience=3 triggered. SWA ran 10 epochs with constant LR 1e-4, followed by BN re-estimation (100 steps, 3,200 images).

Training Details

Training Strategy

¹ Phase 1 uses EarlyStopping with patience=5 on val_accuracy. Ran 25 epochs (maximal), best epoch = 23 (val_accuracy 97.54%). EarlyStopping was not triggered — model kept improving within every 5-epoch window.

² Phase 2 uses EarlyStopping with patience=3 on val_accuracy, followed by 10 SWA epochs (constant LR 1e-4).

Hyperparameters

Regularization Strategy

Dataset

Data Preprocessing

• Normalization: preprocess_input from tf.keras.applications.efficientnet_v2 (ImageNet distribution)
• Input resolution: 320×320 (higher than ImageNet default 224×224 to capture fine-grained architectural details — textures, ornaments, facade patterns)
• Augmentation: Applied to training set only. validation and test sets use clean preprocessing
• Split method: splitfolders.ratio from dataset/, seed=42

Files

Usage

Gradio Space

Python — build_model.py (recommended)

• Custom class definitions (GeMPooling, FocalLoss, DiscriminativeAdamW) with @register_keras_serializable — importing the module registers all custom classes globally, so load_model() works without explicit custom_objects.
• ArchBuildingClassifier — high-level wrapper class with build(), from_weights(), from_keras(), predict(), predict_batch() methods.
• CUSTOM_OBJECTS dict — fallback for explicit custom_objects= in load_model().
• build_model() — backward-compatible function that returns a raw tf.keras.Model.

Note: Filenames below use fine_tuning_swa as an example. The actual best checkpoint filename depends on training results — check the repo for the actual .keras, .weights.h5, and .safetensors filenames.

Python — TF-Lite (fastest inference)

Python — Keras (convenient)

Python — SavedModel (TF Serving)

Python — safetensors (HF standard, cross-framework)

Note: safetensors stores raw weight tensors without architecture metadata. To load, reconstruct the architecture with build_model.py first, then map tensors manually. For most use cases, .weights.h5 (via ArchBuildingClassifier.from_weights()) is simpler and equally clean.

Inference Verification

The 1 misclassification (castle→bridge, 41.9% confidence) is consistent with the 97.92% test accuracy. The 8/8 match confirms TFLite conversion preserves model behavior exactly.

Security Notice (PAIT-KERAS-301)

• What the scanner checks: String-matching of class_name fields in the Keras v3 config against a whitelist of built-in Keras layers.
• Why flagged: The model contains a custom layer (GeMPooling) — a non-standard class name triggers the flag.
• What it does NOT check: The scanner does not analyze the Python code of the custom class, does not look for eval()/exec()/os.system(), and does not detect actual malware.
• Other scanners: VirusTotal, JFrog, HF Picklescan — all clean. Only Protect AI flags this file.

• GeMPooling — Generalized Mean Pooling (Radenovic et al., CVPR 2018). Pure tensor ops: tf.pow, tf.reduce_mean, tf.maximum.
• FocalLoss — Focal Loss (Lin et al., ICCV 2017). Pure tensor ops.
• DiscriminativeAdamW — AdamW subclass with gradient scaling. No file I/O, no network calls, no arbitrary code.

Multi-Format Deployment Guide

Load Examples

Intended Use

• Architectural style classification from building photographs
• Educational tool for architecture recognition
• Research baseline for fine-grained image classification (FGIC)
• Transfer learning experiments on architectural imagery

Limitations

• Trained on Pexels stock photography — performance may differ on user-generated or field photographs
• Limited to 8 architectural classes (barn, bridge, castle, mosque, skyscraper, stadium, temple, windmill)
• Confusion pair analysis found 0 significant pairs (threshold >5%) — all 8 classes are well-distinguished by the model. see confusion_pairs.json for details
• Barn and windmill share 3 cross-class duplicates (0.02% of dataset) — left as-is due to negligible impact
• Inference confidence can be low on atypical examples

Ethical Considerations

• All training images sourced from Pexels.com under the Pexels License (free for commercial use, no attribution required). No copyrighted or personally identifiable images were used.
• The dataset contains only photographs of buildings and structures — no people, faces, or private property are the subject of classification.
• The model reflects the visual distribution of Pexels stock photography, which may over-represent Western and iconic architectural styles and under-represent vernacular or regional architecture.
• The 8 class categories are broad and do not capture the full diversity of world architecture. Results should not be used to make definitive claims about architectural categorization.
• URL pattern filtering during dataset collection explicitly excluded AI-generated art, illustrations, and non-photographic content to ensure authenticity.

Links

• Gradio Space (Live): arch-building-classifier Space
• Dataset Studio: 0xgr3y/arch-building-dataset
• GitHub Repository: arcxteam/building-architectural-image-classifier

References

1. Tan, M., & Le, Q. V. (2021). EfficientNetV2: Smaller Models and Faster Training. ICML 2021. arXiv:2104.00298
2. Radenovic, F., Tolias, G., & Chum, O. (2018). Fine-Tuning CNN Image Retrieval with No Human Annotation. IEEE TPAMI. arXiv:1711.02512
3. Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017). Focal Loss for Dense Object Detection. ICCV 2017. arXiv:1708.02002
4. Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018). Averaging Weights Leads to Wider Optima and Better Generalization. UAI 2018. arXiv:1803.05407
5. Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). mixup: Beyond Empirical Risk Minimization. ICLR 2018. arXiv:1710.09412
6. Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. ICCV 2019. arXiv:1905.04899
7. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. CVPR 2016. arXiv:1512.00567
8. Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How Transferable Are Features in Deep Neural Networks? NeurIPS 2014. arXiv:1411.1792
9. Howard, J., & Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. ACL 2018. arXiv:1801.06146
10. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR, 15(56), 1929–1958. http://jmlr.org/papers/v15/srivastava14a.html
11. Ioffe, S., & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv preprint. arXiv:1502.03167
12. Tarvainen, A., & Valpola, H. (2017). Mean Teachers are Better Role Models: Weight-averaged Consistency Targets Improve Semi-supervised Deep Learning Results. NeurIPS 2017. arXiv:1703.01780
13. Perez, L., & Wang, J. (2017). The Effectiveness of Data Augmentation in Image Classification using Deep Learning. arXiv preprint. arXiv:1712.04621
14. Shanmugam, D., Blalock, D., Balakrishnan, G., Guttag, J., & Sarma, A. (2020). Towards Principled Test-Time Augmentation. ICML 2020. PDF
15. Loshchilov, I., & Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR 2017. arXiv:1608.03983
16. Prechelt, L. (1998). Automatic Early Stopping Using Cross Validation: Quantifying the Criteria. Neural Networks, 11(4), 761–767. https://doi.org/10.1016/S0893-6080(98)00010-0
17. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML 2017. arXiv:1706.04599
18. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. ICCV 2017. arXiv:1610.02391
19. van der Maaten, L., & Hinton, G. (2008). Visualizing Data using t-SNE. JMLR, 9(Nov), 2579–2605. http://jmlr.org/papers/v9/vandermaaten08a.html
20. Hand, D. J., & Till, R. J. (2001). A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning, 45(2), 171–186. https://doi.org/10.1023/A:1010920819831
21. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., ... & Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3), 211–252. arXiv:1409.0575
22. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS 2017. arXiv:1612.01474

Citation