You can deploy this model and start earning money today!

rwightman/
mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3

Name: mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3
Availability: InStock

zero-shot-image-classification

pixparse/cc12m-wds

apache-2.0

Views

No views yet

Model card for mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3

MaMMUT2 with a modern text decoder (RoPE, RMSNorm, SwiGLU, gated attention, qk-norm) and tiktoken r50k vocab at ctx 128. Best moderntext run in the comparison.

Research checkpoint from a B/32-scale architecture/objective comparison on CC12M (CLIP vs MaMMUT2 vs modern text decoder); not a production model.

Model Details

Dataset: pixparse/cc12m-wds
Epochs: 32
Precision: bf16 amp
Weight decay: 0.25
Objective: InfoNCE + 2x caption CE + 1e-5 z-loss
Text decoder: modern (RoPE/RMSNorm/SwiGLU/gated attn), qk-norm, tiktoken r50k vocab 50260, ctx 128, variable-length text
Optimizer: adamw, lr 2e-3
Batch size: 1024

Model Usage

python

import torch
import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:rwightman/mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3')
tokenizer = open_clip.get_tokenizer('hf-hub:rwightman/mammut2-moderntext_ViT-B-32.cc12m-qk-lr2e3')

image = preprocess(pil_image).unsqueeze(0)
text = tokenizer(['a dog', 'a cat', 'a bird'])
with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)

Caption generation:

python

with torch.no_grad():
    tokens = model.generate(image, generation_type='beam_search', seq_len=30)
print(tokenizer.decode(tokens[0].cpu().tolist()))

Model Comparison

run	IN1K	IN-V2	IN-R	IN-Sk	I2T R@1	T2I R@1	CIDEr	BLEU-4	METEOR
clip-siglip	42.72	36.44	54.38	32.08	34.84	25.95
clip-naflex	40.4	34.43	50.53	28.55	34.6	24.25
mammut2 (no z)	37.8	31.98	48.8	27.16	37.52	24.72	10.02	2.43	7.37
mammut2 (+z)	38.36	32.47	49.44	27.12	38.18	24.54	10.04	2.41	7.38
mammut2-naflex	37.54	31.92	48.63	26.28	38.64	24.7	9.52	2.33	7.4
mammut2-mt no-qk lr1e-3	38.07	32.78	50.84	27.28	37.44	25.18	10.96	3.58	7.69
mammut2-mt qk lr2e-3 (this model)	38.37	33.02	50.56	27.18	37.94	24.93	10.85	3.44	7.57
mammut2-mt qk lr2e-3 b4096	40.0	34.21	53.91	29.81	37.7	25.37	12.23	3.91	7.99
mammut2-mt qk lr3e-3	37.05	31.82	48.41	26.03	35.92	24.06	10.29	3.31	7.35

All eight runs: CC12M, 32 epochs, 2x GPU, bf16 amp, wd 0.25, evaluated at epoch 32 with one harness (ImageNet zero-shot w/ OpenAI 80-prompt ensemble; COCO Karpathy-test retrieval, I2T = any-of-5-captions; caption metrics vs the 5 COCO refs, beam search). Key reads:

The CLIP rows are prior runs, not purpose-trained controls: they were the closest available baselines but come from NAdaMuon optimizer experiments (nadamuon @ bs 4096 vs the MaMMUT runs' adamw @ bs ~1024), so cross-family deltas bundle optimizer + batch + loss.
Caption objective -> I2T retrieval: every MaMMUT2 run beats both CLIP runs by 1-4 pt I2T R@1 despite 1/4 the contrastive batch and a plainer optimizer.
Zero-shot ordering tracks batch/loss/optimizer, not architecture -- confirmed by the bs-4096 moderntext run: at matched batch (still adamw) it lands within 0.4 IN1K of clip-naflex and within 0.5 IN-R of clip-siglip. Batch was most of the CLIP lead.
The caption-objective I2T advantage is batch-independent (37.7-38.6 I2T R@1 at both bs 1024 and 4096), while zero-shot, T2I, and caption metrics all scale with batch.
NaFlex vision and 1e-5 caption z-loss are both ~free (within the <=0.6 pt repeat-run noise).
Modern text tower (bundled with r50k tokenizer + ctx 128) at its best LR: IN1K parity with classic, +V2/+IN-R/+T2I, and clearly better caption metrics. qk-norm buys 2x LR headroom rather than direct quality; 3e-3 is past the stable edge even with it.
Caption scores are style-limited: CC12M alt-text register vs COCO references. Absolute CIDEr is low for all runs; relative differences are the signal.