clapkit-coreml — fp16-safe CoreML conversions of LAION CLAP (both encoders)
CoreML conversions of
laion/clap-htsat-unfused
— both towers of the CLAP audio↔text model — converted from the PyTorch source
(
transformers ClapModel) with the campaign's
fp16-survivable numerical guard
discipline, so they are correct on the Apple GPU and Neural Engine, not just CPU.
Consumed by the
clapkit crate of the
coremlit workspace (Rust, sync,
sans-I/O). Each graph emits the final
512-d joint-space embedding pre-L2-norm;
the caller performs the L2 normalization (this keeps the fp16
rsqrt guard class
out of the graph entirely).
Contents
| artifact | form | what it is |
|---|
clap_audio.{mlpackage,mlmodelc} | fp16 | HTSAT audio tower + audio_projection (spectrogram-input) |
clap_text.{mlpackage,mlmodelc} | fp16 | RoBERTa text tower + text_projection |
clap_audio_int8.{mlpackage,mlmodelc} | int8 | audio tower — 8-bit k-means-palettized weights (weight-only; activations fp16) |
clap_text_int8.{mlpackage,mlmodelc} | int8 | text tower — 8-bit k-means-palettized weights (weight-only; activations fp16) |
.mlpackage is the canonical distributable; .mlmodelc is the compiled form the
test suite loads directly. CHECKSUMS.sha256 covers every file. The _int8
siblings are an optional ~2× smaller tier (see "int8 weight-only tier" below);
the fp16 graphs remain the default.
I/O contract (pinned from the artifact metadata)
| encoder | input(s) | output |
|---|
| audio | input_features fp32 [1, 1, 1001, 64] (log-mel) | audio_embeds fp32 [1, 512] (pre-norm) |
| text | input_ids int32 [1, 512], attention_mask int32 [1, 512] | text_embeds fp32 [1, 512] (pre-norm) |
- Audio is 48 kHz mono, a documented deviation from the workspace's 16 kHz
convention (CLAP's native rate). One inference = one fixed 480,000-sample (10 s)
window → the
[1, 1, 1001, 64] mel.
- Text length is fixed at 512 (the model's max). Padding a shorter prompt to
512 with the attention mask reproduces the natural-length embedding exactly
(verified cosine
1.0), because RoBERTa derives positions from input_ids and
the mask zeroes the padding.
The mel frontend lives in Rust (spectrogram-input), by measurement
The audio graph takes the log-mel spectrogram, not raw audio. Riding the
STFT + Slaney-mel + power_to_dB frontend inside the graph was attempted and
rejected on measurement:
- a faithful in-graph STFT reproduces HF's
ClapFeatureExtractor mel only in
float64 (the reference promotes to f64; an in-graph f32 STFT lands 0.55–0.90
cosine away from the correct path end-to-end) — f64 is hostile to an fp16 ANE
graph;
- the
power_to_dB floor amin = 1e-10 is 1680× below fp16's smallest subnormal
(2^-24) — exactly the vanishing-guard class this campaign guards against.
So the mel is a Rust port validated bit-for-bit against textclap's mel.rs
(the frontend oracle). Its parameters: n_fft = 1024, hop = 480, n_mels = 64,
fmin = 50, fmax = 14000, periodic Hann, Slaney scale + Slaney norm,
center=True reflect padding, 10·log10(max(·, 1e-10)), HTSAT input-norm none,
time-major [1001, 64] → reshaped to [1, 1, 1001, 64].
Verification (measured, this conversion)
-
MIL-level fp16-guard audit CLEAN — 55 guard sites total (audio 30, text 25),
every effective floor ≥ 2^-24; no decomposed softmax→log; no unresolved
guards. The text tower's LayerNorm eps = 1e-12 (below fp16 subnormal) is raised
to 2^-24 by the fp16 conversion; audio LayerNorm/BatchNorm eps ≈ 1e-5 survive
as-is. Normalization is out of the graph, so there is no rsqrt/real_div guard.
-
PyTorch fp32 vs CoreML fp32 (CPU) on real inputs: worst cosine 1.00000000
(audio, 10 real clips: music / speech / SFX / ambient) and 1.00000000 (text,
12 varied prompts). The graph conversion is faithful; the HTSAT reshape_mel2img
bicubic resize is reproduced by an exact baked-constant matmul.
-
CoreML fp16 vs fp32, worst cosine per compute unit:
| encoder | ALL | CPU+GPU | CPU |
|---|
| audio | 0.99999573 | 0.99999390 | 0.99996626 |
| text | 0.99994979 | 0.99999733 | 0.99987553 |
fp16 is clean on every placement — fp16 is shipped (no fp32 fallback needed).
int8 weight-only tier (optional, ~2× smaller)
Alongside the fp16 graphs this repo ships an 8-bit weight-only tier of both
towers — clap_audio_int8 / clap_text_int8. Weights are stored 8-bit and
dequantized to fp16 at runtime; activations stay fp16 (identical graph maths,
only the constant weights are compressed). Produced from the shipped fp16
.mlpackages by coremltools post-training compression — no reconversion, same
source weights.
-
Method: 8-bit k-means palettization, per-tensor
(coremltools.optimize.coreml.palettize_weights, nbits=8, mode="kmeans",
weight_threshold=2048). Chosen over linear-symmetric int8
(linear_quantize_weights) by measurement: on the text tower linear int8
costs up to 0.43% cosine (worst 0.99571 vs fp16) while k-means costs
0.13% (worst 0.99873); on audio the two are close (0.99965 vs 0.99971).
k-means wins on both towers and is decisive on text, so it is the shipped _int8
tier. Tiny bias / LayerNorm-gain tensors (< 2048 elements) stay fp16; one RoBERTa
attention-mask fill constant (±inf) is likewise left uncompressed.
-
int8-vs-fp16 embedding cosine (identical inputs, CPU; the conversion
verification set — 10 real clips + 12 varied prompts including CJK — plus the
committed golden_mel fixture):
| tower | worst | mean |
|---|
| audio | 0.99970584 | 0.99982787 |
| text | 0.99873250 | 0.99965990 |
Well inside a 0.5% cosine budget on both towers. (int8-vs-fp32-source worst:
audio 0.99967, text 0.99897 — barely further from the PyTorch source than fp16
itself, which is 0.99997 / 0.99988.)
-
Zero-shot ranking unchanged. The end-to-end gate (a ~192 s speech clip → 20
windows → aggregate → 4-anchor zero-shot score) returns the identical ranking
on int8 as on fp16: top label "This is a sound of a person speaking" at logit
8.88 (fp16 8.82), same order for music / dog / rain, same margins.
-
Placement unchanged. Cross-compute-unit agreement holds (audio 0.99998, text
0.99994 — ≥ the 0.9999 band on All / CPU+NE / CPU+GPU / CPU). Audio still fails
ANECCompile() and falls back to GPU/CPU; text still compiles for the ANE — the
palettized weights change neither.
-
Size (weight.bin bytes, fp16 → int8):
| tower | fp16 | int8 | ratio |
|---|
| audio | 60,562,432 | 30,409,920 | 1.99× |
| text | 251,665,792 | 125,746,688 | 2.00× |
Compiled .mlmodelc: audio 58 MB → 29 MB, text 240 MB → 120 MB.
Same toolchain pin as the fp16 conversion (coremltools 9.0; scikit-learn provides
the k-means step). The int8 tier is a pure recompression of the pinned fp16
artifacts, so its provenance and licensing are exactly those of the fp16 graphs
below.
Placement guidance (measured, never marketed)
- text: compiles for and runs on the ANE/GPU/CPU; fp16-clean on all.
- audio: the HTSAT graph currently fails ANE compilation (
ANECCompile())
and falls back to GPU/CPU — still fp16-clean there. Consumers should not assume
ANE placement for the audio tower; clapkit selects the compute unit and does
not assert ANE.
Toolchain (pinned)
coremltools 9.0 · torch 2.5.1 · transformers 5.14.0 · numpy 1.26.4 · python 3.11.15.
Source revision: laion/clap-htsat-unfused@8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a.
Upstream provenance & licensing
These are derivative conversions: the same LAION CLAP weights, re-emitted as
CoreML graphs (projection heads in-graph, L2-norm out, mel frontend externalized).
License note. Attribution to laion/clap-htsat-unfused is provided as
required. The LAION CLAP checkpoints are commonly attributed as CC-BY-4.0
(the position taken by the consuming project and reflected in the front-matter),
while the current upstream HF model card declares apache-2.0. Both licenses
require attribution, which this repository provides; downstream users should
honor whichever the upstream author intends. If you are the upstream author and
want changes to this redistribution, please open a discussion.
