TASTE2-8B-EN with Instruction-Following Task Vector
This repository contains
slm.pt — the speech language model (SLM) backbone of
TASTE2-8B-EN, merged with an instruction-following (IF) task vector derived from Qwen2-7B-Instruct.
Model Overview
TASTE2-8B-EN is an English speech language model that generates speech tokens auto-regressively, conditioned on text instructions. Its SLM backbone is a Qwen2-7B-based language model trained on English speech data.
This model merges IF capability into the TASTE-EN backbone without any SFT, using task vector arithmetic.
Merge Method
Let:
- $\theta_{\text{TASTE-EN}}$ — dense LLM weights extracted from the original TASTE2-8B-EN
slm.pt
- $\theta_{\text{Qwen2-7B}}$ — original Qwen2-7B base weights
- $\theta_{\text{Instruct}}$ — Qwen2-7B-Instruct weights
Step 1: Compute TIES-sparsified IF task vector
$$\tau = \theta_{\text{Instruct}} - \theta_{\text{Qwen2-7B}}$$
$$\hat{\tau} = \text{TopK}_{30%}(\tau) \quad \text{(keep top 30% by magnitude, zero out the rest)}$$
$$\delta_{\text{IF}} = 0.5 \times \hat{\tau}$$
Step 2: Apply task vector to TASTE-EN backbone
$$\theta_{\text{merged}} = \theta_{\text{TASTE-EN}} + \delta_{\text{IF}}$$
| Parameter | Value |
|---|
| Merge method | TIES |
| Density | 0.3 |
| Weight | 0.5 |
| SFT after merge | None |
Files
| File | Description |
|---|
slm.pt | Merged SLM backbone (18.7 GB). Drop-in replacement for the original slm.pt in TASTE2-8B-EN. |
To run inference, place this slm.pt alongside the other TASTE2-8B-EN model files (flow.pt, hift.pt, llm.pt, speech_tokenizer_v2.onnx, campplus.onnx, etc.) and use the standard TASTE inference pipeline.
Citation
If you use this model, please cite the original TASTE paper:
1@article{taste,
2 title={TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling},
3 author={Tseng, Liang-Hsuan and Chen, Yi-Chang and Lee, Kuan-Yi and Shiu, Da-Shan and Lee, Hung-yi},
4 journal={arXiv preprint arXiv:2504.07053},
5 year={2025},
6 url={https://arxiv.org/abs/2504.07053}
7}
