Detection head architectures on frozen EUPE-ViT-B features. The backbone is frozen throughout; only the head trains. The premise behind the repository is that a strong modern ViT encoder already carries enough semantic and spatial structure to support detection with a lightweight decoder on top, and that the detection head's architecture — not the backbone — is where the interesting design trade-offs live. Every result here is obtained from features extracted once and cached; training iterates over those cached tensors, which makes it possible to run architecture studies in minutes rather than the days required for end-to-end backbone training.

The headline artifact is a 2,975,067-parameter split-tower detection head that reaches 42.64 mAP on COCO val2017 (42.71 under soft NMS), beating the standard FCOS baseline (16.14M parameters, 41.0 mAP) by +1.64 at 18.4 percent of its head parameter budget. Small-object mAP is 22.3 against FCOS's 19.4 (+2.9). This is the head shipped in phanerozoic/argus. It replaces FCOS's learned feature pyramid with a zero-parameter cofiber decomposition of the backbone patch features: four frequency-separated scale bands at strides 16, 32, 64, and 128 (produced by iteratively subtracting the downsampled-then-upsampled component of the feature map), plus a stride-8 level synthesized by a single transposed convolution. Each level is processed by separate classification and regression towers (five 3×3 ConvGN blocks followed by four depthwise residual blocks at 160 hidden channels, with weights shared across levels and top-down lateral connections). Classification is cosine similarity of a Linear(160, 768) projection against CLIP ViT-L/14 multi-prompt text embeddings of the 80 COCO class names. Regression uses exponentiated LTRB distances with a learned per-level scale. Training is 16 epochs at 768-pixel input with ATSS target assignment, flip augmentation, EMA, and PC-initialized cls_project, followed by a 3-epoch partial fine-tune that updates only the classification calibration layers (cls_project, cls_bias, logit_scale) while the towers and cofiber path stay frozen.

The repository also contains a standard FCOS detector (16.14M parameters, 41.0 mAP) as the canonical reference baseline. It trains identically on the same cached features and uses the same eval pipeline, so the 42.64-vs-41.0 comparison is apples-to-apples — both heads read the same frozen backbone output, both are evaluated with pycocotools hard NMS at IoU 0.5. Anyone who wants to reproduce the comparison, or to see how far a standard FCOS head gets on modern frozen ViT features, can take the reference off the shelf.

Alongside the shipped picker and the FCOS reference sits a library of seventeen experimental architectural scaffolds under heads/. Each is a single head.py file implementing a forward pass with no trained checkpoint — hypothesis-stage designs that formulate detection in unconventional ways: wavelet decomposition in place of FPN, optimal-transport label assignment, compositional patch assembly, tropical-semiring operations, corner-pair relational reasoning, and others. A driver script (arena.py) instantiates and trains any of them against the cached backbone features, so trying a new formulation is a one-command operation. The scaffolds are recorded so alternative formulations are not lost, and so readers looking for detection ideas have somewhere to browse.

The full parameter-versus-mAP scaling curve for the cofiber line of work (from 105-parameter minimal circuits up through the current 2.98M picker at 42.64 mAP) is hosted in the separate phanerozoic/cofiber-detection repository, along with the machine-checked Rocq/HoTT proof of the decomposition's exactness, analytical closed-form heads, and synthesizable Verilog circuit implementations. This repository keeps the top-performing checkpoint and the experimental scaffold library; the research lineage lives in the sibling repo.

Reference Results (COCO val2017)

Head	Parameters	mAP	mAP@0.50	mAP@0.75
Split-tower, 160h, 768px + cls_calib fine-tune (16 ep + 3 ep partial)	2,975,067	42.64	65.70	45.10
Split-tower, 160h, 768px (16 ep, step 104k, pre-calib)	2,975,067	42.49	65.57	44.89
Split-tower, 160h, 640px (16 ep, step 100k, pre-resolution-scale)	2,975,067	41.15	63.99	43.83
Split-tower + CLIP ViT-B/32 text-aligned, 192h (8 ep, 640px)	4,164,699	25.95	39.13	28.92
Split-tower with multi-scale decomposition, 192h (8 ep, no text align)	4,068,954	24.6	37.1	27.0
Baseline FCOS (simple feature pyramid)	16,138,074	41.0	64.8	43.2

Eval protocol: per-class hard NMS (IoU 0.5), score threshold 0.05, top-100 per image, pycocotools COCO val2017. Canonical script: eval_coco_map.py. A --soft-nms flag runs an additional linear-decay soft-NMS pass for comparison.

The top row is the current picker and the head shipped in phanerozoic/argus. At 2.98M learnable parameters it reaches 42.64 mAP on COCO val2017 (42.71 under soft NMS), passing the 16.14M FCOS baseline (41.0 mAP) by +1.64 while using 18.4 percent of its head parameter budget. Small-object mAP is 22.3 (vs FCOS 19.4, +2.9). Soft NMS gives 42.71 mAP, 65.73 @0.50, 45.33 @0.75.

The FCOS baseline uses a simple feature pyramid that synthesizes five scale levels (P3 at stride 8 through P7 at stride 128, each with 256 channels and GroupNorm) from the backbone's stride-16 spatial features, followed by two shared four-layer convolutional towers (one for classification, one for box regression plus centerness) and three 1×1 prediction heads. It trains in eight epochs at 640-pixel input. The configuration is standard and the implementation is direct; the point of having it in this repository is to provide a clean, comparable reference.

The split-tower head differs from the FCOS baseline in three ways: the multi-scale structure comes from a zero-parameter cofiber decomposition of the backbone features rather than from a learned FPN; the classification and regression towers use separate weights throughout rather than sharing a tower; and the classifier is a linear projection into frozen CLIP text-embedding space followed by cosine similarity against the 80 COCO class-name embeddings, replacing a learned per-class weight matrix. The classifier projection is a single Linear(hidden, text_embed_dim) without bias; class prototypes are the L2-normalized CLIP text-encoder outputs for the 80 COCO class names; a learned scalar temperature and per-class bias calibrate the cosine scores.

The current checkpoint is the end point of a six-step lineage that lifted the same architecture from 36.88 to 42.64 mAP at roughly constant parameter count:

Step	Change	mAP (hard NMS)
baseline	160h 5+4 towers, FCOS assignment, no aug, 640px	36.88
+ ATSS + horizontal flip aug	stride-scaled pseudo-IoU assignment, 50% flip	40.06
+ EMA + PC init + CLIP ViT-L/14 8-prompt	EMA decay 0.9998, cls_project columns initialized from SVD of text_embed, switch from ViT-B/32 single-prompt to ViT-L/14 8-prompt-average text embeddings	40.89 (live) / 40.96 (EMA)
+ 16-epoch schedule, late-training sweep	double the schedule, select peak from 11-point sweep over steps 80k-117k	41.15 (live) / 41.27 (soft NMS)
+ 768-pixel training	same recipe at 48×48 backbone grid instead of 40×40	42.49 (step 104k peak)
+ 3-epoch cls_calib fine-tune	partial fine-tune updating only `cls_project`, `cls_bias`, `logit_scale` at lr 1e-4 (towers and cofiber path frozen)	42.64 / 42.71 (soft NMS)

The 4.16M / 25.95 mAP row is a prior-generation picker (192-hidden, 8 epoch, CLIP ViT-B/32 single-prompt text embeddings), retained for continuity. The jump from 25.95 to 42.64 is training-recipe and training-resolution: ATSS target assignment, horizontal flip augmentation, EMA, PC-initialized cls_project, ViT-L/14 multi-prompt text embeddings, a doubled schedule, the switch from 640-pixel to 768-pixel training input, and a 3-epoch classification-calibration fine-tune. Hidden width went down from 192 to 160 and the head still shrank while gaining 16.7 mAP. The recipe changes live in train_split_tower_5scale.py; eval JSONs for every shipping checkpoint sit next to the weights in the same directory.

Analytical single-class person detector

A probe of how far a detection head can go with zero gradient descent on the same frozen backbone. The best closed-form configuration reaches 2.33 AP / 10.3 AP@0.50 on COCO val2017 for the person class at ~105K learnable parameters, all derived from simple statistics of the cached training features (no SGD, no training loop):

Fisher LDA classifier — 768-dim direction computed in closed form from pooled covariance of person-positive vs. background features across 85K positives and 51M negatives. Bayes-calibrated bias incorporates the ~1:600 class-imbalance prior of person-positive patches in COCO.
15-anchor aspect grid — 5 scale quantile bins × 3 fixed aspect ratios (0.4, 1.0, 2.5) guarantees landscape-orientation coverage even though COCO person training data skews portrait. A per-scale ridge-regressed 15-way classifier selects the anchor at each patch location.
Per-anchor fine refinement — 15 independent 770-dim ridge regressors each predict residuals (dx, dy, d_log_w, d_log_h) relative to their assigned anchor shape. Splitting the regression into per-anchor subproblems avoids the averaging failure that broke earlier global-regression variants.

AP@0.75 is effectively zero (0.2): the 15 discrete anchor shapes plus linear residual correction produce boxes precise enough for loose localization but not for strict IoU thresholds. Generative-classifier variants (per-class Gaussian mixtures via Mahalanobis distance) and soft anchor weighting were tested and both underperformed the discriminative-classifier + argmax configuration — a clean negative result showing Fisher LDA beats GMM for this task, and hard anchor selection beats soft weighted mean.

Checkpoint at heads/cofiber_threshold/analytical_person_v3j/head_v3j_canonical.pth; build-and-evaluate script at _analytical_person_v3j.py. Total wall time to reproduce: ~3 minutes (statistics pass + closed-form solves + full val eval). The 80-class analytical scaling of this approach is in the separate cofiber-detection repository referenced below.

Cofiber decomposition (separate repository)

A large parallel line of work on this backbone replaces the FCOS feature pyramid with cofiber decomposition: a zero-parameter multi-scale operation that iteratively subtracts the downsampled-then-upsampled component of a feature map to produce frequency-separated scale bands. Given backbone features f, the decomposition produces f − upsample(avgpool(f)) as a high-frequency band, then recurses on the low-frequency remainder. The three bands that result are pairwise orthogonal in frequency content, which lets a lightweight head attend to them separately without needing a learned FPN to mediate between scales.

The decomposition has been machine-checked in Rocq/HoTT. The proof frames average pooling and bilinear upsampling as an adjoint pair whose counit gives a short exact sequence in a semi-additive category; the cofiber bands are the kernels of the resulting projections, and any input is uniquely expressible as a sum of its bands. The practical consequence is that the multi-scale construction which typically costs 11M parameters in an FPN is free — no parameters, no training, no FLOPs beyond pooling and interpolation.

This has accumulated enough work around it to warrant its own home. The sibling repository phanerozoic/cofiber-detection contains:

Closed-form analytical heads derived from least-squares regression on the decomposed features (zero training, 70K parameters, 1.6 mAP)
Trained neural heads at multiple scales, from 70K parameters (5.2 mAP) up to the 3.85M-parameter split-tower included here (20.3 mAP) and 4.27M-parameter variants with top-down lateral connections (19.9 mAP)
INT8 threshold-logic heads using Heaviside step functions (92K parameters, 5.9 mAP) and their pruned variants down to 46K nonzero parameters
Synthesizable Verilog circuit implementations, including a 93-parameter person image classifier that achieves 99.8% recall on COCO val images
Evolutionary dimension search (GPU-batched, 200 generations per second) that identifies informative feature subsets; the evolved 10-dimension person-detection circuit outperforms a greedy 100-dimension circuit at 10× fewer gates
Sheaf-cohomology regression features (directional feature differences at spatial boundaries, interpretable as Čech 1-cocycles)
The Rocq/HoTT proof of decomposition exactness
Full training and evaluation scripts

Readers interested in any of the above — parameter-efficient detection, formalized multi-scale operations, hardware circuits, evolutionary feature selection, topological regression features — should go there. This repository keeps the top-performing head as a comparison point but does not attempt to mirror the full research lineage.

Experimental Architectural Scaffolds

The seventeen scaffolds under heads/ each explore a distinct hypothesis about how detection could be formulated on frozen ViT features. None has been trained to convergence; the point is to record architectural options rather than to benchmark them:

Scaffold	Core idea
`adaptive_query`	Learned query vectors that adaptively attend to backbone patches, producing detections as attention outputs rather than per-location predictions
`cascade_pool`	A cascade of pooling operations to synthesize multi-scale features without an FPN
`centernet`	CenterNet-style keypoint detection: predict object centers as a heatmap, then regress size and offset at center locations
`compression`	Head operating on compressed backbone features, testing how aggressively features can be reduced before detection quality collapses
`curvature`	Object boundaries as high curvature of the feature manifold; detection as curvature estimation
`depth_fusion`	Joint detection and depth estimation sharing features, exploiting the alignment between depth gradients and object boundaries
`feature_graph`	Detection as a graph problem over patch tokens, with edges indicating co-occurrence or spatial adjacency
`mutual_attention`	Cross-attention between patch features to model inter-object relationships before detection
`optimal_transport`	Label assignment via optimal transport between ground-truth boxes and prediction locations
`patch_assembly`	Compositional object construction from overlapping patch detections, inspired by self-assembly tile systems
`prototype`	Replacing learned classification weights with fixed class prototypes (text-embedded or image-mean)
`relational_corners`	Corner-pair detection: predict corner heatmaps, then pair them via learned embeddings
`scale_classify`	Explicit scale classification as an auxiliary head to produce scale-consistent boxes
`sparse_query`	Sparse query mechanism to limit prediction locations and reduce background noise
`threshold_prototype`	Prototype classification with a threshold gate
`tropical`	Tropical-semiring (min-plus) operations in the head
`wavelet`	Wavelet-based multi-scale decomposition as an alternative to FPN and to cofiber decomposition

Each is a one-file architectural commitment. Most will not beat the FCOS baseline; some may suggest directions that do. The arena script makes training any of them a one-command operation against the cached features.

FCOS Variants

Three FCOS-family heads are present as controlled variants on the baseline, each differing from the reference in one isolated way:

heads/baseline_fcos/ — the full simple-feature-pyramid FCOS head at 16.14M parameters, the 41.0-mAP reference.
heads/slim_fcos/ — a reduced-parameter variant with narrower towers and fewer FPN channels. It exists to help locate how much of the baseline's capacity is load-bearing.
heads/hook_fcos/ — an FCOS head that reads intermediate backbone features via forward hooks rather than only the final spatial output. This follows the pattern used by the DPT depth decoder in the multi-task Argus model, where intermediate ViT blocks (2, 5, 8, 11) carry information that the final block does not expose; this variant tests whether the same access pattern improves detection.

Arena

arena.py and multi_domain_arena.py are lightweight driver scripts that instantiate any head from the heads/ directory, train it for a configurable number of epochs on cached COCO features, and evaluate with pycocotools. The multi-domain version extends evaluation across the twenty RF100-VL cross-domain datasets in addition to COCO, to test whether a head's performance generalizes beyond the benchmark it was tuned on.

Shared Infrastructure

losses/ contains the FCOS-style focal loss for classification, GIoU loss for boxes, BCE for centerness, and a CenterNet-style focal-loss variant for keypoint heads.
utils/ contains the feature caching code (shard format, manifest, shard loader), the decoder that converts per-location predictions to COCO-format boxes, and the evaluation wrapper around pycocotools.
eval_coco_map.py is the standalone evaluation entry point; it can be pointed at any saved checkpoint.
train_split_tower_5scale.py and eval_split_tower_5scale.py reproduce the split-tower results above.

Backbone

All results use EUPE-ViT-B with letterbox padding. The backbone produces 768-channel spatial features at stride 16 — a 40×40 grid at 640-pixel input (used by the FCOS reference baseline and the earlier-generation cofiber pickers) or a 48×48 grid at 768-pixel input (used by the shipped 2.98M picker). The backbone has 86M parameters and is frozen; features are cached once per resolution before any head training begins. The framework is backbone-agnostic: any ViT-scale encoder producing a consistent spatial grid can be substituted by replacing the feature extraction step, and the head library will run without modification. EUPE-ViT-B is the default because its multi-teacher distillation produces features that work well across detection, segmentation, depth, and correspondence, which makes it a sensible proving ground for heads that might eventually share a backbone with other tasks.

References

Tian, Shen, Chen, He. FCOS: Fully Convolutional One-Stage Object Detection. ICCV 2019.
Li, Mao, Girshick, He. Exploring Plain Vision Transformer Backbones for Object Detection. ECCV 2022.
Zhou, Wang, Krähenbühl. Objects as Points. arXiv:1904.07850, 2019.

License

Apache 2.0. Users are responsible for complying with the license terms of whatever backbone they substitute for EUPE-ViT-B.

phanerozoic/detection-heads

Detection Heads

Contents