A 183.5M-parameter image encoder built from scratch — adaptive tokenization, hierarchical concept tokens, and continuous 2D positional encoding. Not a CLIP/ViT fine-tune.
RGB image, 384×384 (any multiple of 16 supported natively)
Output
512 adaptive tokens (768-dim) + one pooled 768-dim image embedding
Trained
From random init on ~20M image–text pairs, 18,000 steps, 2× H200
Precision
bfloat16
Benchmark
Result
ImageNet zero-shot Top-1 / Top-5
37.2% / 66.8%
Flickr30K Image→Text R@1 / R@5 / R@10
42.9% / 72.7% / 80.6%
Flickr30K Text→Image R@1 / R@5 / R@10
42.2% / 70.0% / 79.4%
CIFAR-100 linear probe
75.0%
Measured at 384×384 with token_budget=512, matching training. Accuracy trails
same-size peers — see Limitations and Evaluation
for why, and for the higher-k retrieval results where the gap closes.
Downstream Results — Vision-Language Model
The encoder is frozen and used as-is inside a bilingual (English + Korean)
vision-language model: mkd-hika/keural-mid-vlm-bilingual.
Only a projector (8.9M) and LoRA adapters (5.0M) are trained on top; the decoder is
Qwen2.5-7B-Instruct.
A vision-language model can score respectably while barely using the image, because
many questions are answerable from language priors alone. Each benchmark below is
therefore run twice — once normally, once with the visual tokens removed and
everything else held constant. The difference is what this encoder contributes,
and it is the only figure the language model cannot produce on its own.
Benchmark
With this encoder
Blind
Encoder contributes
POPE accuracy (n=2,000)
70.50%
44.55%
+25.95
POPE F1 (n=2,000)
70.59%
23.46%
+47.13
VQAv2 strict (n=5,000)
53.04%
39.83%
+13.21
MME total, 14 categories (n=2,374)
1258.9
812.0
+446.9
MMBench (n=4,377)
60.20%
—
—
Held-out perplexity (KO / EN)
2.15 / 2.99
—
—
The POPE yes-ratio is 0.492 against a gold ratio of 0.500, so the model genuinely
discriminates on object-existence questions rather than defaulting to one answer —
accuracy alone is not interpretable without that check. The blind VQAv2 score of
39.83% lands where language-prior performance is expected to, which is corroborating
evidence that the measurement is sound.
Notably, the encoder reaches these numbers while scoring 37.2% zero-shot on ImageNet.
Zero-shot accuracy measures image–text alignment; the 75.0% CIFAR-100 linear probe
is the better indicator of representation quality for downstream use, because a
trained projector relearns the alignment from scratch.
Model Description
Keural Mid is a mid-scale vision encoder that maps an image to a single 768-dim embedding aligned to a text embedding space, plus a variable-length sequence of semantically-typed tokens. It is trained from random initialization (no CLIP/ViT/DINO backbone) with a combination of sigmoid contrastive learning, hierarchical concept alignment, saliency regularization, and dual-teacher knowledge distillation.
It introduces three mechanisms not found together in existing encoders (CLIP, SigLIP, DINOv2):
Innovation
What it does
ATB — Adaptive Token Budget
Content-aware token allocation. A learned saliency map drives a differentiable Gumbel-Top-K sampler, spending more tokens on complex regions and fewer on simple backgrounds.
HCT — Hierarchical Concept Tokenization
Every token carries a semantic level — global, region, or detail — via learned level embeddings and level-specific constructors.
Continuous 2D-RoPE
Rotary position encoding from actual normalized (cx, cy) coordinates plus a log-area scale bias, so variable resolutions and token scales are handled natively.
Intended Uses & Limitations
Intended uses
Image feature extraction — a global pooled embedding for retrieval, clustering, or as a frozen backbone for downstream heads (e.g. linear probing, detection, captioning front-ends).
Research into adaptive tokenization, hierarchical token typing, and resolution-elastic position encoding.
Limitations
Accuracy trails same-size peers. Zero-shot ImageNet Top-1 is 37.2%, versus 68.6% for CLIP ViT-B/16 (~86M params) and 76.7% for SigLIP ViT-B/16 (~93M params) — both smaller than this model. This is attributed to a training-budget gap (18K steps on ~20M pairs, vs. peers trained on 12M-400M+ pairs over far more steps) rather than an architecture ceiling: training loss was still declining at the final step. See Evaluation.
Saliency mechanism is real but token placement is still settling. The learned saliency map produces stable, object-tracking contours on images with a clear dominant subject. However, the discrete tokens it samples continued shifting meaningfully until very late in training (only ~50% overlap with the final checkpoint's token positions as of step 13-14K, reaching ~73% by step 16K) — exact token placement had not fully converged even at the final checkpoint.
Domain/bias. Trained on web image–text pairs; it inherits the coverage and biases of that distribution and is English-text aligned.
Not a safety-filtered model. No content moderation or de-biasing has been applied.
Checkpoints. The repo root holds the final model (step 18,000). Earlier
checkpoints are available as subfolders for studying training dynamics:
checkpoint-9000, checkpoint-10000, checkpoint-15000, checkpoint-18000
— load one with AutoModel.from_pretrained(..., subfolder="checkpoint-15000").
Note: pass token_budget=512 explicitly. Training always used a fixed
budget of 512 tokens; leaving token_budget unset falls back to
resolution-based auto-scaling (e.g. 1,152 tokens at 384px), which the model
was not trained on. Evaluation showed this mismatch moves results by <1
point, so it's a minor effect, but 512 is the methodologically correct
setting to match training.
Semantic level per token: 0=global, 1=region, 2=detail.
spatial_metadata
(B, N, 4)
(cx, cy, scale_w, scale_h), normalized to [0, 1].
saliency_scores
(B, N)
Per-token importance in [0, 1].
attention_mask
(B, N)
Valid-token mask.
Architecture
The figure above is the complete specification: (a) the tokenizer stack
(CNN stem → saliency → ATB → HCT), (b) the spatial transformer block and
continuous 2D-RoPE, (c) the output contract, dual-teacher distillation, and
the training objective.
The pooled image embedding is the [POOL] token after the final RMSNorm (no separate projection head). For contrastive training it is aligned against a trainable projection of a frozen CLIP text encoder.
2.87 total (down from 21.56 at step 0), still declining at completion
Loss.L_total = L_primary + λ_hct·L_hct + λ_sal·β_sal·L_saliency + L_distill, where L_primary and L_hct are sigmoid image↔text losses (on the pooled embedding and on the mean of global tokens respectively), L_saliency is an anti-collapse + total-variation regularizer on the saliency map, and L_distill is cosine distance to the two frozen teachers.
Training loss curves
Distillation Teachers
Teacher
Params
Embed dim
Projection to 768
SigLIP-SO400M
400M
1152
Linear(1152 → 768)
InternViT-300M
300M
1024
Linear(1024 → 768)
Both teachers are frozen and used only during training; they are not required for inference.
Evaluation
Evaluated on the final checkpoint with token_budget=512 (exactly matching training).
Benchmark
Result
ImageNet zero-shot Top-1
37.2%
ImageNet zero-shot Top-5
66.8%
Flickr30K Image→Text R@1 / R@5 / R@10
42.9% / 72.7% / 80.6%
Flickr30K Text→Image R@1 / R@5 / R@10
42.2% / 70.0% / 79.4%
CIFAR-100 linear probe Top-1
75.0%
Benchmark comparison
Flickr30K Recall@k
Multi-axis comparison
Context. Against its own distillation teachers, Keural Mid retains ~45% of SigLIP-SO400M's zero-shot ImageNet accuracy (83.1%) and ~48-56% of its Flickr30K retrieval R@1, after 18K steps on ~20M pairs vs. teachers trained on billions of pairs. Against same-parameter-class peers (CLIP ViT-B/16, SigLIP ViT-B/16 — both smaller than this model), accuracy trails by roughly 2x. Both gaps are consistent with a training-budget shortfall rather than an architecture ceiling: loss was still declining at the final step. A compute/data-matched fixed-grid baseline is the recommended next experiment to isolate the ATB architecture's own contribution from this gap.
The R@1 gap narrows sharply at higher k
The headline R@1 numbers understate retrieval quality. Broken out by k:
Flickr30K Recall@5
Flickr30K Recall@10
On Text→Image R@5, Keural Mid (70.0%) outperforms CLIP ViT-B/16 trained on WIT-400M (57.2%), and at R@10 it leads 79.4% vs. 68.0% — despite that baseline having seen ~20x more training data. The correct-image is usually retrieved within the top few results even when it isn't ranked first, which suggests the embedding space is broadly well-organized and that the R@1 shortfall is largely a ranking-precision issue rather than a representation-quality one. (Note the CC12M-trained CLIP variant still leads at these k, so this is not a uniform win — it is specific to the WIT-400M baseline on the text→image direction.)
Adaptive tokenizer — saliency & token placement
Saliency evolution across training
The learned saliency map produces sharp, stable contours that track each image's dominant subject, visible from mid-training through the final checkpoint. Quantitatively, the continuous saliency map stabilizes fast (cosine similarity to the final checkpoint reaches ~0.83-0.90 within the first few thousand steps), but the discrete sampled token positions are considerably less stable — only ~50% overlap with the final checkpoint's tokens by step 13-14K, jumping to 73% at step 16K. Exact token placement was still moving late into training even where the saliency field itself looked converged.
