Audio-into-CLIP alignment for text–image–audio retrieval. Edgebind maps audio into the existing embedding space of a frozen CLIP model instead of training a joint encoder from scratch, so audio, text and images become directly comparable by cosine similarity in one 512-d space.

Research artifact for "Edgebind: Towards Edge-Compatible Audio-into-CLIP Alignment for Text-Image-Audio Retrieval" (CAISc 2026).

Note on the datasets metadata field. This card intentionally omits it. Clotho v2.1 has no canonical publisher-owned dataset repository on the Hub — a search returns only third-party mirrors and the unrelated ClothoAQA task. The training code downloads Clotho directly from Zenodo, so tagging a mirror would misstate provenance. The authoritative source is linked in prose under Training data.

Model description

Component	Detail
Text & image encoders	OpenCLIP ViT-B/32, `hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K` — fully frozen, never updated
Audio encoder	Audio Spectrogram Transformer, `MIT/ast-finetuned-audioset-10-10-0.4593`
AST frozen	Patch embeddings, position embeddings, encoder layers 0–8
AST trained	Encoder layers 9, 10, 11 and the final layernorm
Pooling	Mean of the first two AST output tokens (CLS and distillation)
Projection head	`Linear(768, 1024) → LayerNorm(1024) → ReLU → Dropout(0.3) → Linear(1024, 512)`
Temperature	Learnable `logit_scale`, initialized to `log(1 / 0.07)`
Output	512-d, L2-normalized, shared with CLIP text and image embeddings
Objective	Symmetric InfoNCE (cross-entropy over in-batch negatives in both directions, averaged)
Training data	Clotho v2.1, development split

Only the AST upper layers, the projection head and the temperature receive gradients. Because the CLIP towers are untouched, text and image embeddings produced by this model are identical to stock OpenCLIP ViT-B/32 — the alignment is carried entirely by the audio branch.

The pooling choice is worth stating explicitly, since it is easy to assume otherwise: audio features are the average of tokens 0 and 1, not the CLS token alone. Using CLS alone is a separate ablation in the training notebook.

Intended use and limitations

Intended use: text-to-audio and audio-to-text retrieval over a local corpus you control — embed a set of audio clips once, then rank them against free-text queries by cosine similarity. Because the CLIP image tower is frozen and shared, images can be embedded into the same space and searched with the same text queries.

This is a research prototype, not a production model. Specifically:

Single dataset. Trained and validated only on Clotho v2.1, which is small (a few thousand clips) and skewed toward everyday environmental and ambient sound. Behaviour on speech, music, or domain-specific audio is uncharacterized.
Single training run, no seed variance. One run of 20 epochs. The training code sets no random seed, and caption sampling, crop offsets and SpecAugment masks are all stochastic, so run-to-run variance has not been measured. Treat any single reported number as one sample, not a mean.
No CPU, quantized, or on-device benchmarking. The model was trained and run under CUDA mixed precision on a single NVIDIA T4. Latency, memory, and accuracy under CPU inference, quantization, distillation, or mobile/embedded runtimes have not been evaluated. The released checkpoint is unquantized fp32. Despite "edge-compatible" in the paper title — which refers to the design motivation for reusing a frozen ViT-B/32 backbone — nothing here establishes that this model is edge-ready or deployable on constrained hardware. Do not treat it as such.
Fixed 10.24 s window. Audio is cropped or zero-padded to exactly 163,840 samples at 16 kHz. Longer recordings are truncated, not chunked; content outside the window is invisible to the model.
Inherited bias. The text and image behaviour is entirely that of LAION-2B-trained OpenCLIP ViT-B/32 and carries its biases unchanged.

Out of scope: audio captioning or generation (there is no decoder), speaker or speech recognition, and any safety-, surveillance-, or identity-related classification.

How to use

The checkpoint is a state_dict for the composite module defined in the training notebook, so you must reconstruct that module — CLIP submodule included — before loading. The snippet below mirrors the notebook's own model definition and its evaluation-time preprocessing path.

python

import numpy as np
import torch
import torch.nn as nn
import torchaudio
import torchaudio.transforms as T
from huggingface_hub import hf_hub_download
from transformers import ASTModel, AutoProcessor
import open_clip

MODEL_NAME = "hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K"
AST_NAME = "MIT/ast-finetuned-audioset-10-10-0.4593"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"


class OpenCLIP_AST_Model(nn.Module):
    def __init__(self, embedding_dim=512):
        super().__init__()
        self.ast = ASTModel.from_pretrained(AST_NAME)
        self.clip_model, _, self.image_preprocess = open_clip.create_model_and_transforms(
            MODEL_NAME, pretrained=None
        )
        self.audio_projection = nn.Sequential(
            nn.Linear(self.ast.config.hidden_size, 1024),
            nn.LayerNorm(1024),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(1024, embedding_dim),
        )
        self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))

    def forward_audio(self, input_values):
        out = self.ast(input_values)
        # Mean of CLS + distillation tokens, matching training.
        feats = (out.last_hidden_state[:, 0] + out.last_hidden_state[:, 1]) / 2
        return self.audio_projection(feats)


weights = hf_hub_download("harryfrz/edgebind", "edgebind_v1.1")

model = OpenCLIP_AST_Model().to(DEVICE)
# The notebook loads with a plain torch.load. On torch >= 2.6 the weights_only=True
# default is appropriate for a pure tensor state_dict; pass weights_only=False if
# your torch version raises on it.
model.load_state_dict(torch.load(weights, map_location=DEVICE))
model.eval()

ast_processor = AutoProcessor.from_pretrained(AST_NAME)
tokenizer = open_clip.get_tokenizer(MODEL_NAME)

SAMPLE_RATE = 16000
TARGET_SAMPLES = 163840  # 10.24 s at 16 kHz


def embed_audio(path):
    waveform, sr = torchaudio.load(path)
    waveform = waveform.mean(dim=0) if waveform.shape[0] > 1 else waveform.squeeze(0)
    if sr != SAMPLE_RATE:
        waveform = T.Resample(sr, SAMPLE_RATE)(waveform)

    n = waveform.shape[0]
    if n < TARGET_SAMPLES:
        waveform = torch.nn.functional.pad(waveform, (0, TARGET_SAMPLES - n))
    elif n > TARGET_SAMPLES:
        start = (n - TARGET_SAMPLES) // 2  # centre crop, as at evaluation time
        waveform = waveform[start:start + TARGET_SAMPLES]

    inputs = ast_processor(waveform, sampling_rate=SAMPLE_RATE, return_tensors="pt")
    with torch.no_grad():
        emb = model.forward_audio(inputs["input_values"].to(DEVICE))
    return emb / emb.norm(dim=-1, keepdim=True)


def embed_text(prompts):
    with torch.no_grad():
        emb = model.clip_model.encode_text(tokenizer(prompts).to(DEVICE))
    return emb / emb.norm(dim=-1, keepdim=True)


def embed_image(pil_image):
    with torch.no_grad():
        x = model.image_preprocess(pil_image).unsqueeze(0).to(DEVICE)
        emb = model.clip_model.encode_image(x)
    return emb / emb.norm(dim=-1, keepdim=True)


# --- retrieval over a local corpus ---
corpus = ["clip_a.wav", "clip_b.wav", "clip_c.wav"]
index = torch.cat([embed_audio(p) for p in corpus], dim=0)   # [N, 512]

query = embed_text(["waves hitting the shore"])              # [1, 512]
scores = (query @ index.T)[0]                                # cosine similarity

for rank in scores.argsort(descending=True):
    print(f"{corpus[rank]}  {scores[rank].item():.4f}")

open_clip.create_model_and_transforms(MODEL_NAME, pretrained=None) still downloads pretrained weights: the hf-hub: prefix resolves the checkpoint from the Hub, and pretrained=None only means no additional named tag is applied. Those weights are then overwritten by the clip_model.* entries in the state dict.

What the checkpoint file contains

edgebind_v1.1 (955 MB, no file extension). The training code saves with torch.save(model.state_dict(), ...) and reloads with a strict model.load_state_dict(torch.load(path, map_location=device)). That means:

It is a raw state_dict, not a training checkpoint — no optimizer state, no epoch counter, no scheduler or scaler state, no embedded config or metrics.
It stores the complete module, not only the trained tensors. Key prefixes are ast.* (the whole AST, frozen layers 0–8 included), clip_model.* (the entire frozen OpenCLIP ViT-B/32 image and text towers), audio_projection.*, plus the scalar logit_scale.
That is roughly 239 M parameters in fp32 (AST ≈86 M + CLIP ≈151 M + projection ≈1.3 M), consistent with the 955 MB file size.
Loading is strict, so the module must be rebuilt exactly — including the CLIP submodule, which is why the snippet above instantiates it.

This description is derived from the notebook's save/load code and the file size, not from opening the file. The tensor keys and dtypes have not been enumerated directly. If you need that confirmed, load it and inspect .keys().

Evaluation

No results table is published in this card. The training notebook was committed with all cell outputs stripped, and it contains no code that computes R@1/R@5/R@10, MedR, MeanR or mAP@10, no CLAP baseline, and no frozen-AST ablation. Transcribing metrics from any other source would not be verifiable against this repository, so none are reproduced here.

For the reported text-to-audio and audio-to-text results on the Clotho v2.1 evaluation split, and the baseline and ablation comparisons, see the CAISc 2026 paper.

The notebook does include a qualitative check — top-3 retrieval for five hardcoded prompts — and one ablation, CLS-only pooling, described below.

Training details

Training data

Clotho v2.1, development split, obtained from Zenodo record 4783391. The training code downloads clotho_captions_development.csv and clotho_audio_development.7z from that record directly. Each audio file carries five human-written captions; one is sampled at random per example per epoch during training.

Preprocessing

Downmix to mono, resample to 16 kHz.
Crop or zero-pad to 163,840 samples (10.24 s): random crop during training, centre crop at evaluation.
Log-mel features via AutoProcessor for MIT/ast-finetuned-audioset-10-10-0.4593.
SpecAugment, training only: FrequencyMasking(freq_mask_param=24) and TimeMasking(time_mask_param=40).

Hyperparameters

Setting	Value
Optimizer	AdamW, `weight_decay=0.05`
Epochs	20
Batch size	64 (gradient accumulation steps = 1, so effective batch = 64)
LR — AST layers 9–11	5e-6
LR — AST final layernorm	5e-6
LR — projection head	2e-4
LR — logit scale	2e-4
Schedule	`CosineAnnealingLR`, `eta_min=1e-6`, stepped per optimizer step
Precision	CUDA mixed precision (`torch.amp.autocast` + `GradScaler`)
Loader	`drop_last=True`, `num_workers=4`

Compute

Single NVIDIA T4. The training cell's recorded execution window spans 50 min 25 s for all 20 epochs plus audio-embedding export. That figure comes from notebook cell execution timestamps, not from a printed training log.

Ablation included in the code

CLS-only pooling — identical in every other respect, but uses last_hidden_state[:, 0] instead of the mean of tokens 0 and 1. It trains to a separate checkpoint. No metrics for it are present in the notebook.

A note on naming

Some artifacts and in-code comments use the internal name "Sage-Embed" (for example, # Verified Sage-Embed v1.1 train/freeze configuration). This refers to the same model as Edgebind. The v1.1 suffix on the checkpoint filename corresponds to that internal versioning.

Citation

bibtex

@inproceedings{edgebind2026,
  title     = {Edgebind: Towards Edge-Compatible Audio-into-CLIP Alignment for
               Text-Image-Audio Retrieval},
  author    = {Harikrishna C},
  booktitle = {CAISc},
  year      = {2026}
}

The repository records no full author list, DOI, or page numbers; the author field above is taken from commit metadata and should be completed before use.

License

CC BY 4.0.

harryfrz/edgebind

Edgebind