Model card for naflexclap_base_pf8_pt20_moderntextp.240m

Methodology

• Standalone CLAP suite: audio uncapped (batch-max padding); classification rows show default prompt ("This is a sound of {c}.") and best-of-sweep; retrieval is A2T/T2A = text-retrieval / audio-retrieval R@5.
• MAEB(beta)-30: --batch_size 16, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, uncapped except BirdCLEF capped at 60 s for comparability (A 0.604 was pre-capped; B/C/D recomputed at 60 s).
• Precision: fp32 weights+activations, no autocast, TF32 matmuls (the MAEB encoder default). Not a precision artifact.

1. Architecture, params, geometry

• A vs D — text tower only (bidir+MAP+qk-norm vs causal+EOS+bias), everything else equal. Cleanest.
• A vs C — audio trunk (768/12 ungated GELU vs 512/20 gated GELU) + geometry (pf8/pt16 vs pf4/pt20).
• A vs B — confounded 3 ways: pt16→pt20, 160M→240M samples, audio GELU→SwiGLU. Not attributable to one.

2. Standalone CLAP suite

Classification is prompt-sensitive: default "This is a sound of {c}." is pathological for some checkpoints (D UrbanSound 0.495→0.709 best; C 0.456→0.775 best). Best-prompt is the fairer signal. GTZAN best uses "This is a {c} song."/"a {c} song"; UrbanSound/ESC-50 best vary per model.

‡ UrbanSound rows use the full danavery/urbansound8K (8732 clips), matched data for zero-shot + probe.

Probe = official UrbanSound8K protocol: supervised logreg on audio embeddings, 10-fold CV on the predefined fold splits (train 9 / test 1, ×10, averaged) — literature-comparable. All four land at 0.838–0.856 (1.8 pt spread) despite zero-shot ranging 0.46–0.80, so the entire UrbanSound zero-shot spread is text-alignment / prompt, not audio quality: e.g. D probes 0.8441 (above B and C) yet its default zero-shot was 0.494 (EOS-pooling text misalignment; audio is fine). (A naive random 5-fold inflates the probe to ~0.93 via same-recording leakage — do not use it; the UrbanSound8K authors explicitly warn against reshuffling.)

¶ VGGSound = SINGLE-label, metric is top-1/top-5 accuracy (309 classes; not mAP). Source mteb/VGGSound test (9,888 clips, audio + integer label, ready 309-class ClassLabel vocab — no reconstruction); audio uncapped; VGGSound prefers a template (bare {} worst 0.22–0.26); B/C/D peak at "This is a sound of {c}.", A at "the sound of {c}"; ordering B > C > D > A (A's text tower aligns differently, ~6 pt best-to-best). Harness validated against LAION-CLAP (laion/clap-htsat-unfused through the same pipeline = 0.234 top-1, ≈ its published VGGSound zero-shot ~0.26).

3. MAEB(beta)-30 — full per-task (main_score)

4. Audio-representation probes (supervised, prompt-independent)

• VoxPopuli gender — facebook/voxpopuli test, all 16 language configs (n = 16,991, 69% male, 1133 speakers). Speaker-disjoint GroupKFold(5) (same speaker ⇒ same gender, else a random split leaks), class_weight=balanced; read straight from the cached parquet (no datasets-arrow rebuild).
• UrbanSound8K — the official 10-fold probe from §2 (full danavery 8732, predefined folds), shown here for the audio-representation view.

Key findings

1. Text tower (A vs D, clean isolation): bidirectional+MAP > causal+EOS — MAEB 0.4545 vs 0.4276 (−2.7 pt). The loss concentrates on rich-text retrieval (CommonVoice −0.20, Fleurs −0.20, JamAltLyric −0.15, Clotho, SpokenSQuAD, MInDS14): EOS pooling compresses the whole caption into the last token; MAP keeps the full sequence retrieval needs. Even/better on short-label classification (SpeechCommands +0.04, CREMA_D +0.04). Causal+EOS is also more prompt-fragile (its weak default-prompt zero-shot undersells it).
2. Longer training (A vs B, +confounds): helps the CLAP suite, flat on MAEB. B is best on ESC-50/AudioCaps/ Clotho/BirdCLEF but ties A on MAEB (0.4533 vs 0.4545) — two big drops (SpeechCommands −0.17, VoxPopuliGenderClustering −0.19) cancel broad gains; excluding those two, B is +1.2 pt over A on the other 28. B's SpeechCommands −0.17 is checkpoint-specific (D, also causal-free pt16, is +0.04 there).
3. mediumd audio trunk (A vs C): −2.1 pt MAEB, but C is 17% smaller (139.7M) and the gap is task-dependent (wins BirdCLEF/GTZAN, loses speech/temporally-fine). No robust pt16-vs-pt20 effect once prompts are matched.
4. No pt20 UrbanSound weakness and no fill/precision artifacts — UrbanSound clips are ≤4 s (~21 tokens), too short for pt to matter; per-class pt20 gains on jackhammer; pad-mode (floor/silence/repeat) moves SpeechCommands <0.003; MAEB runs fp32.

Result provenance (persistent)

• Eval scripts (under those dirs): clapv2_cls_sweep.py, gtzan_sweep.py, birdclef_capped.py (argv model/ckpt/cap/out), urban_perclass.py, urban_durations.py, speechcmd_padmode.py, urban_full.py (full-8732 danavery: zero-shot sweep + official 10-fold probe in one embedding pass; urban_probe10.py = probe-only; urban_probe.py = old random-5-fold, superseded); vox_gender_probe.py (VoxPopuli gender, speaker-disjoint GroupKFold; VOX_LANGS env selects configs; results in voxgender.log = 5-lang, voxgender16.log = full 16-lang).

Appendix — pf8 vs pf4 (frequency tokens): confounded, but a directional signal