The acoustic encoder of microsoft/VibeVoice-1.5B, fully separated into a standalone voice-embedding model: 24 kHz audio in, one 64-dimensional speaker vector out.

344M parameters (the encoder alone — no language model, no diffusion head, no decoder)
64-d embeddings, mean-pooled over causal-VAE latent frames at 7.5 Hz
Deterministic: the encoder's distribution mean is used, never a sample — the same clip always produces the same vector
Loads in seconds, runs at >100× realtime on a modest GPU slice (~2.7 GB at fp32)

How it was made

Upstream VibeVoice pairs this encoder with a decoder as a reconstruction VAE feeding a 1.5B LM. The weights here are the 276 model.acoustic_tokenizer.encoder.* tensors, copied bit-for-bit (bfloat16, exactly as stored) out of the source checkpoint and renamed to encoder.*.

One encoder covers the whole family

The 1.5B and 7B (Large) VibeVoice checkpoints ship the same acoustic encoder. Measured against a 7B export (fp16 storage): 111/276 tensors bit-identical, the rest differ only by bf16-vs-fp16 storage rounding (≤5×10⁻⁶ of tensor max, concentrated in elements small enough to hit fp16's subnormal range); embeddings of identical clips agree to ~2×10⁻⁵ relative and speaker-separation metrics are identical (d′ +3.61, AUC 0.997 for both). So this export is the voice embedder for either model size. The modeling code is likewise mechanically extracted from upstream modular_vibevoice_tokenizer.py with the decoder classes removed — only VibeVoiceEmbedModel (construction, pooling, padding handling) is new. Provenance, including the exact source revision, is recorded machine-readably under vibevoice_embed_metadata in config.json.

Does a reconstruction VAE make a speaker embedder?

Measured on a controlled set — 4 speakers × 4 utterances, all different text, each clip cloned from one fixed reference so identity is pinned (24 same-speaker pairs, 96 different-speaker pairs):

representation / metric	d′	AUC
mean-pooled, cosine on unit vectors	+3.61	0.997
mean-pooled, L2 on raw vectors	+2.80	0.975
‖norm gap‖ alone	+0.15	0.562
purpose-built speaker embedder (CosyVoice, 192-d), cosine	+3.47	0.990

So mean-pooled VibeVoice latents separate speakers on par with a purpose-trained speaker-verification embedder on this set, despite coming from an encoder trained for reconstruction.

Honest caveats:

Every test clip is synthetic output of one TTS engine, so recording channel is matched in a way that flatters the encoder. Not yet validated on real, diverse-channel diarized audio.
No metric achieved a positive worst-case margin — same/different distributions overlap slightly, so this supports ranking and clustering, not a single global accept/reject threshold.
Pooling is mean-only for a measured reason: std-pooling carries no speaker information here (std vectors of different clips have cosine 0.97–0.98), and concatenating it degrades the embedding (its norm is 3–4× the mean's).

Distance metric

Vectors are returned unnormalised (raw preserves information; normalising is one line). But the per-clip magnitude is noise — scored alone it is at chance (AUC 0.562) — so ranking by L2 on raw vectors is measurably worse than cosine. L2-normalise before indexing, or configure your vector store for cosine. On unit vectors, cosine, Euclidean and dot product produce identical rankings.

Usage

python

import torch, torchaudio
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "lemuriandezapada/VibeVoice-Embed",
    torch_dtype=torch.float32,   # recommended: pooling is precision-sensitive
    trust_remote_code=True,
).eval().to("cuda")

wav, sr = torchaudio.load("clip.wav")
wav = wav.mean(0, keepdim=True)  # mono
if sr != model.config.sampling_rate:
    wav = torchaudio.functional.resample(wav, sr, model.config.sampling_rate)

with torch.no_grad():
    out = model(wav.to("cuda"))

embedding = out.pooler_output[0]            # (64,), float32, unnormalised
unit = torch.nn.functional.normalize(embedding, dim=0)  # for cosine indexing
frames = out.last_hidden_state              # (1, frames, 64) latents at 7.5 Hz

Batching variable-length clips

The encoder is causal, so right-padding cannot corrupt a clip's real frames — but the frames emitted for the padding must not be averaged in. Pass padding_mask and the model pools mask-aware:

python

batch = torch.nn.utils.rnn.pad_sequence([wav_a[0], wav_b[0]], batch_first=True)
mask = torch.zeros_like(batch, dtype=torch.bool)
mask[0, : wav_a.shape[-1]] = True
mask[1, : wav_b.shape[-1]] = True
out = model(batch.to("cuda"), padding_mask=mask.to("cuda"))

A clip embedded in a batch equals the same clip embedded alone.

Serving

serving/openai_embeddings_server.py is a self-contained OpenAI-compatible /v1/embeddings server (FastAPI): audio goes in the input field as base64 or a data: URI, optionally "normalize": true for unit vectors.

python serving/openai_embeddings_server.py --model lemuriandezapada/VibeVoice-Embed --port 8080

There is no vLLM plugin, deliberately: this is a pure convolutional encoder — no attention, no KV cache, no tokens — so vLLM's engine has nothing to schedule for it, and at >100× realtime the serving bottleneck is transport, not compute.

Repository layout

model.safetensors — encoder weights, bfloat16, keys encoder.*
config.json — encoder hyperparameters (verbatim from the source checkpoint), auto_map for trust_remote_code, and vibevoice_embed_metadata provenance
configuration_vibevoice_embed.py / modeling_vibevoice_embed.py — standalone modeling code, no dependency on the VibeVoice package
serving/openai_embeddings_server.py — reference server

Upstream references

Code: https://github.com/microsoft/VibeVoice (community fork: https://github.com/vibevoice-community/VibeVoice)
Base model: https://huggingface.co/microsoft/VibeVoice-1.5B
Related export from this workspace: VibeVoice-ASR-awq-int4

Notes

This is a derivative export of the upstream checkpoint's acoustic encoder, not a fine-tune; no weights were modified.
Base model licensing and usage terms follow the upstream VibeVoice release (MIT).