This repository contains all model checkpoints, experiment notebooks, and results from a thesis studying whether pixel-wise soft-weighting can improve smoke segmentation when supervision comes from noisy SAM-generated pseudo-labels.

Short answer: No — under multi-seed evaluation, no weighting variant reliably outperforms the plain pseudo-label baseline (M2). All six methods achieve 0.164–0.185 mIoU with overlapping standard deviations. The single-run apparent range (0.060–0.220 mIoU) was an artefact of random initialisation, not a reliable property of the loss designs.

Architecture

Component	Detail
Model	DeepLabV3+
Encoder	MobileNetV2 (depthwise separable convolutions)
Parameters	~5.8 M
Input size	400 × 400 px
Output	2-class Softmax (smoke / background)
Pre-training	Smoke5K dataset (clean pixel labels)
Fine-tuning	IJmond pseudo-labels (167 images, SAM-generated)
Test set	IJmond human annotations (2,074 images, never seen during training)
Optimizer	Adam · lr=1e-4 · wd=1e-4 · cosine annealing · 30 epochs · batch 8

Methods

Six loss formulations are compared. All share the same DeepLabV3+ network at inference; differences are only in how per-pixel cross-entropy losses are aggregated during training.

Notation

$P_{\text{label}}(i)$ — Softmax probability of the pseudo-labeled class at pixel $i$ (detached from the gradient graph)
$P_{\text{local}}(i)$ — 5×5 average-pool of $P_{\text{label}}$ (spatial neighbourhood mean)

Method table

ID	Name	Weight $W(i)$	Loss formula
M2	Plain pseudo-labeling (baseline)	1	$\frac{1}{N}\sum_i \ell_i$
M3	Hard confidence threshold	$\mathbf{1}[\max_c P_c(i) \geq 0.8]$	normalised masked CE
M1-Conf	Confidence only	$P_{\text{label}}(i)$	$\frac{\sum W_i \ell_i}{\sum W_i + \varepsilon}$
M1-Cons	Consistency only	$1 - \|P_{\text{label}}(i) - P_{\text{local}}(i)\|$	same
M1-Both	Confidence × Consistency	$P_{\text{label}}(i) \times (1 - \|P_{\text{label}}(i) - P_{\text{local}}(i)\|)$	same
M4	Mean Teacher (EMA)	—	$\mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{MSE}}(\text{student}, \text{teacher})$, $\alpha=0.99$

Key design choice: $P_{\text{label}}$ is detached from the computation graph before being used as a weight. Without detach, the model can minimise the loss by driving $P_{\text{label}} \to 0$ (zeroing the weight) instead of improving predictions.

Results

Primary evidence — Multi-seed evaluation (3 seeds × 6 methods)

Mean ± 1 std over 3 independent training runs (seeds 0, 1, 2). Error bars show seed standard deviation. Threshold = 0.5.

Method	mIoU	Dice	Precision	Recall
M2 (Plain Pseudo)	0.1847 ± 0.019	0.3116 ± 0.027	0.4054 ± 0.026	0.2564 ± 0.046
M3 (Hard Threshold 0.8)	0.1810 ± 0.005	0.3065 ± 0.007	0.4146 ± 0.033	0.2445 ± 0.019
M1-Conf (Confidence only)	0.1843 ± 0.024	0.3107 ± 0.035	0.4471 ± 0.036	0.2382 ± 0.031
M1-Cons (Consistency only)	0.1780 ± 0.028	0.3016 ± 0.040	0.4073 ± 0.027	0.2447 ± 0.060
M1-Both (Conf × Cons)	0.1726 ± 0.023	0.2940 ± 0.033	0.4826 ± 0.091	0.2141 ± 0.030
M4 (Mean Teacher)	0.1637 ± 0.025	0.2808 ± 0.037	0.4810 ± 0.045	0.2023 ± 0.045

No weighting variant reliably outperforms M2. All differences lie within seed standard deviation.

Notable cross-seed patterns that are reliable:

M3 has the lowest seed variance (σ = 0.005): binary keep/discard is more robust to initialisation than soft weighting.
M4 has the highest mean Precision (0.481) but the lowest Recall (0.202) across all three seeds: the EMA teacher acts as a conservative regulariser that suppresses uncertain detections.

Per-seed breakdown

Click to expand full per-seed mIoU table

Method	Seed 0	Seed 1	Seed 2	Mean	Std
M2	0.2057	0.1800	0.1685	0.1847	0.019
M3	0.1836	0.1756	0.1837	0.1810	0.005
M1-Conf	0.1992	0.1561	0.1974	0.1843	0.024
M1-Cons	0.2060	0.1776	0.1506	0.1780	0.028
M1-Both	0.1985	0.1565	0.1628	0.1726	0.023
M4	0.1876	0.1652	0.1383	0.1637	0.025

Full metrics (mIoU, Dice, Precision, Recall) per seed: results/multiseed/per_seed_results.csv

Threshold sensitivity

Threshold sweep 0.3 → 0.7 on standalone checkpoints (single run). Method rankings under the standalone run are not reproduced in multi-seed evaluation; the sweep confirms the chosen threshold of 0.5 is not a cherry-picked operating point.

Label budget (RQ2 sub-experiment)

Each method re-trained at 25 / 50 / 75 / 100% of the 167-image training pool (single seed, fixed split).

Method	25% (41 imgs)	50% (83 imgs)	75% (125 imgs)	100% (167 imgs)
M2	0.139	0.153	0.189	0.180
M3	0.151	0.153	0.176	0.142
M1-Conf	0.088	0.089	0.133	0.153
M1-Cons	0.136	0.152	0.183	0.178
M1-Both	0.088	0.089	0.130	0.162

M1-Cons tracks M2 closely across all budget levels. M1-Conf and M1-Both are most unstable at very small budgets (normalised-sum denominator can collapse when batch contains few confident pixels).

Deployment Benchmarks (RQ3)

All M1 variants and M2/M3 share the same deployed network — weighting only affects the training loss, not the model architecture. M4 also deploys as a single network (student only).

Inference latency

Batch size	CPU latency	CPU FPS	GPU latency	GPU FPS
1	152 ms	6.5	10 ms	99.9
4	843 ms	4.7	22 ms	178
8	1671 ms	4.8	45 ms	179

5 warm-up + 20 timed runs. CPU: Apple M-series. GPU: Tesla P100.

Training-step overhead vs M2

Method	CPU overhead	GPU overhead
M2	—	—
M3	−8.1%	+0.4%
M1-Conf	−6.9%	+1.5%
M1-Cons	~−1.0%	~+1.9%
M1-Both	−0.8%	+1.9%
M4	+24.5%	+29.3%

M4's overhead comes from a second forward pass (teacher) + EMA parameter update every step. CPU fluctuations for M3/M1-Conf are within timing noise.

Qualitative Analysis

Three representative prediction patterns from the 2,074-image test set.
Overlays show raw Softmax smoke probability; pixels with P < 0.05 are hidden.

Precision-contrast example — M2 over-predicts, weighting variants stay focused

M2 (purple) generates a large false-positive blob covering industrial structures. M1-Conf (red), M1-Cons (orange), and M1-Both (teal) assign substantially lower probabilities to the same non-smoke region.

Positive-detection example — all methods agree on genuine smoke

All four methods consistently localise the chimney plume with similar spatial extent. Weighting variants do not suppress genuine smoke signals when pseudo-labels are unambiguous.

Consistency-effect example — M1-Conf and M1-Cons diverge

M1-Conf (red) assigns moderate probability to a diffuse region near the chimney top. M1-Cons (orange) suppresses this prediction because the local 5×5 neighbourhood average deviates from the pixel's own confidence — illustrating how consistency weighting can reduce ambiguous false positives but may also suppress genuine detections.

40 full-resolution figures (fp / tp / cons categories) are in figures/qualitative/.

How to Load a Checkpoint

python

import torch
import segmentation_models_pytorch as smp

def load_model(ckpt_path, device="cpu"):
    model = smp.DeepLabV3Plus(
        encoder_name="mobilenet_v2",
        encoder_weights=None,
        in_channels=3,
        classes=2,
    ).to(device)
    model.load_state_dict(torch.load(ckpt_path, map_location=device))
    model.eval()
    return model

# Example: load M2 multi-seed seed-0 checkpoint
from huggingface_hub import hf_hub_download
ckpt = hf_hub_download(
    repo_id="Minggg0518/ijmond-smoke-segmentation",
    filename="checkpoints/multiseed/M2_seed0_best.pth",
)
model = load_model(ckpt)

Run inference

python

import numpy as np
import torch
import torch.nn.functional as F

# Load a 400×400 image (.npy format as used in IJmond dataset)
image_np = np.load("your_image.npy").astype(np.float32)          # H×W×3
t = torch.from_numpy(image_np).permute(2, 0, 1).unsqueeze(0) / 255.0
t = F.interpolate(t, size=(400, 400), mode="bilinear", align_corners=False)

with torch.no_grad():
    logits = model(t)                                              # 1×2×400×400
    prob_smoke = F.softmax(logits, dim=1)[0, 1]                   # 400×400
    pred_mask = (prob_smoke > 0.5).int()                           # binary, threshold=0.5

Repository Structure

├── checkpoints/
│   ├── multiseed/          # 18 checkpoints — 6 methods × 3 seeds (PRIMARY)
│   ├── standalone/         # 6 standalone reference checkpoints
│   └── pretrained_smoke5k.pth
├── experiments/
│   ├── 00_handbook.ipynb   # Unified experiment handbook (all chapters)
│   ├── 01_training.ipynb
│   ├── 02_qualitative_analysis.ipynb
│   ├── 03_effectiveness_evaluation.ipynb
│   ├── 04_label_budget_analysis.ipynb
│   └── 05_deployability_cost.ipynb
├── results/
│   ├── multiseed/          # per_seed_results.csv · summary_mean_std.csv
│   ├── standalone/         # standalone evaluation CSVs + figures
│   ├── budget/             # label budget CSVs + figures
│   └── cost/               # inference + training overhead CSVs
└── figures/
    ├── qual_precision_contrast.png
    ├── qual_positive_detection.png
    ├── qual_consistency_effect.png
    └── qualitative/        # 40 full-resolution comparison figures

Key Takeaways

Multi-seed evaluation is essential. A single training run produced apparent extremes (M1-Cons 0.220, M1-Conf 0.060) that did not reproduce across seeds. The true cross-seed range is only ~0.012 mIoU.
No weighting variant is reliably better than plain pseudo-labeling (M2) under the tested conditions (167-image fine-tuning set, Smoke5K pre-training, 30-epoch cosine annealing).
M1-Conf has a latent confirmation-bias failure mode. When the model drifts background-biased, $W = P_{\text{label}}$ gates out the corrective gradients, and the normalised-sum amplification accelerates convergence to a degenerate minimum. This occurred once (mIoU 0.060) but did not reproduce in 3-seed evaluation.
M4 (Mean Teacher) costs +25–30% training compute per step with no consistent accuracy gain — a poor accuracy-cost trade-off on this dataset.
All M1 variants are free at inference (same model, same speed). They are safe to try even when compute budget is tight.

Citation

Wang Mingjia (2026). Pixel-wise Soft Weighting for Smoke Segmentation under
Noisy Pseudo-labels: A Multi-seed Ablation on IJmond Industrial Data.
Master's Thesis, Vrije Universiteit Amsterdam.
Supervisor: Yen-Chia Hsu.