Views
No views yet
Qwen/Qwen3-Embedding-4B (frozen, downloaded from HuggingFace).openai/whisper-large-v3, last encoder layer (layer 32/32; frozen).mlp3_residual architecture — Linear → LayerNorm → GELU → Dropout → residual block → Linear,
hidden_dim=512, joint_dim=256) mapping the frozen text/audio embeddings into a shared 256-d
space, plus a small learned gender-embedding table (male/female/unknown, dim 16, concatenated
onto the text embedding before projection) and the InfoNCE logit scale/bias.checkpoints/qwen3_whisper_last_seed{0-4}_best.pt), matching
the paper's mean±std-over-5-seeds methodology — any of them can be used with infer.py; they are
equivalent for practical use. To reproduce the exact number reported for a specific seed in the
paper's tables, use the corresponding seedN checkpoint.infer.py downloads them automatically from HuggingFace on first run.1pip install -r requirements.txt
2python infer.py --audio clip.wav --text "тихо прошептала, едва сдерживая слёзы"1python infer.py \
2 --audio clip.wav \
3 --text "тихо прошептала, едва сдерживая слёзы" \
4 --gender unknown \
5 --checkpoint checkpoints/qwen3_whisper_last_seed0_best.pt \
6 --device cuda[-1, 1]:cosine similarity: 0.1823--gender defaults to unknown — this matches the setup used for zero-shot evaluation on
external datasets in the paper (reader gender is generally not known/applicable at inference
time); pass male/female explicitly if you want to condition on a specific target voice.infer.py is intentionally self-contained: it does not import anything from the main sdclap
repository, only reimplements the minimal inference-time logic (see inline comments) — this
keeps it runnable with just pip install -r requirements.txt and a checkpoint file.infer.py) — MIT. Checkpoint weights — MIT (trained parameters only, no third-party
copyrighted content embedded).