CanViT-B/16 — ImageNet-1k LP-FT at canvas grid 8
CanViT-B fine-tuned end to end for ImageNet-1k classification
at an 8×8 canvas
grid, from the ImageNet-21k pretrained flagship. It is the canvas-8 counterpart
of
canvitb16-add-vpe-finetune-g128px-s512px-in1k-2026-04-06-nnx,
which was fine-tuned at canvas 32.
CanViT reads a scene through a sequence of 128 px glimpses, accumulating them
into a persistent scene-wide latent canvas. Classification reads the recurrent
CLS token. See the base checkpoint's card for the architecture.
Why a canvas-8 model exists
canvas_grid is an inference-time choice for CanViT: one learnable initial
canvas patch is broadcast at rollout start, so the same weights serve any
resolution. That holds for the pretrained model. It was not established for
LP-FT, where every parameter moves and the fine-tuning geometry could in
principle be baked in. This checkpoint is the arm that measures it.
The canvas grid is deliberately not recorded in config.json, because the
weights are not tied to it. What this model is, is a set of weights whose
fine-tuning ran at 8.
Measured — ImageNet-1k validation
Full ImageNet-1k val, 50,000 images. Policy C2F, T = 21, scene 512 px, glimpse
128 px, canvas grid 8, --mode finetuned. Top-1 %, mean over
n = 8
independent policy seeds with 95% percentile bootstrap intervals over runs
(10,000 resamples). Evaluated with
CanViT-eval at
232ab11 on an
RTX 4090.
| glimpses seen | t | Top-1 | 95% CI |
|---|
| 1 (full scene) | 0 | 80.54 | — |
| 2 | 1 | 82.21 | [82.16, 82.25] |
| 3 | 2 | 83.08 | [83.04, 83.12] |
| 4 | 3 | 83.67 | [83.64, 83.70] |
| 5 | 4 | 84.12 | [84.09, 84.14] |
| 11 | 10 | 84.21 | [84.19, 84.23] |
| 15 | 14 | 84.24 | [84.21, 84.26] — best |
| 21 | 20 | 84.18 | [84.15, 84.22] |
Accuracy rises 3.58 points over the first five glimpses and then moves 0.12
points across the remaining sixteen. C2F's first viewpoint is the deterministic
full scene, so t = 0 is identical in every repeat and has no interval.
The interval covers viewpoint-order variation under a stochastic policy on this
one checkpoint. It does not cover training-seed variance.
Training
Initialized fused_probe from
canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02, then
all parameters trained — backbone, Canvas Attention and head — which is the
LP-FT protocol, not a frozen probe.
| |
|---|
| steps | 100,080 (terminal), 25,620,480 examples |
| warmup | 25,000 steps |
| global batch | 256 (64 per device × 4) |
| optimizer | AdamW, lr 2.5e-5, weight decay 1e-4, grad clip 1.0 |
| label smoothing | 0.1 |
| rollout | T = 4, full BPTT, cross-entropy at every timestep |
| geometry | scene 512 px, glimpse 128 px, canvas grid 8 |
| hardware | one TPU v6e-4 |
| seed | 0 |
The only deviation from the archived flagship LP-FT recipe is
--recipe.canvas-grid 8. Every other value above is that recipe's default.
Provenance
| |
|---|
| trainer | CanViT-NNX-GCP, JAX/Flax NNX |
| launcher run | 019fbe69ecc97220816472f52a2d5a6f (organon v3) |
| source digest | 33d822573c24630ddf1c72847843df01f150e570872daaf32e6ff8b189dfb484 |
| checkpoint step | 100,080 of 100,080 — the terminal step, not a selected one |
| export | NNX → torch safetensors, gated by scripts/verify_export_oracle.py |
model.safetensors md5 | 1a0bd3dd734e315709cb7e0f80ddf274 |
The export restores the orbax checkpoint as host numpy and asserts every model
leaf is covered, so a missing or renamed parameter fails the export rather than
silently producing a partly-initialized model.
Loading
1from huggingface_hub import hf_hub_download
2
3repo = "canvit/canvitb16-add-vpe-finetune-g128px-s512px-c8-in1k-2026-08-01-nnx"
4weights = hf_hub_download(repo, "model.safetensors")
5config = hf_hub_download(repo, "config.json")
config.json carries
n_classes: 1000 and the architecture fields
(
rw_stride,
n_canvas_registers,
enable_vpe, …). Consumed by
CanViT-PyTorch; pass the canvas grid
you want at inference.
Scope
Fine-tuned and evaluated on static scenes. Numbers above are ImageNet-1k
only — this checkpoint has no segmentation or detection result, and a
canvas-8 model is not the configuration the CanViT paper reports.