You can deploy this model and start earning money today!

maikezu/
ST-QE-SpeechLLM-SpTxt-FT

Views

No views yet

ST-QE-SpeechLLM-SpTxt (Fine-tuned)

A fine-tuned LoRA adapter for Qwen2.5-Omni-7B for quality estimation of speech translation using both source audio and source text. Takes a source audio file, source text, and hypothesis translation and returns a quality score.

Speech translation models are increasingly capable of preserving speech-specific information (e.g. speaker gender, prosody, and emphasis), yet evaluation metrics remain blind to such phenomena. We meta-evaluate both text- and speech-based quality estimation metrics on two contrastive datasets targeting gender agreement and prosody, and find that both fall short, even when given direct access to the speech signal. We then train SpeechCOMET, a family of quality estimation models with speech encoders, and evaluate a state-of-the-art SpeechLLM as a judge. Both match or exceed text-based COMET on standard quality estimation, but neither consistently assesses speech-specific phenomena. We identify three causes: (1) speech-specific features are not reliably preserved in current encoders, (2) models tend to ignore the source signal, and (3) quality estimation training data contains too few relevant examples.

Part of the SpeechCOMET model family | Paper: Why We Need Speech to Evaluate Speech Translation (Züfle et al., 2026) | Code: github.com/MaikeZuefle/speechCOMET

Installation

Follow the installation instructions for Qwen2.5-Omni.

Usage

Merge the adapter with the base model first using the LlamaFactory merge script, then:

python

import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info

model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    "maikezu/ST-QE-SpeechLLM-SpTxt-FT", torch_dtype="auto", device_map="auto"
)
processor = Qwen2_5OmniProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B")

system_prompt = (
    "You are an evaluator. Given the source and/or audio and a translation, "
    "respond with only a single float score between 0 and 1 indicating translation "
    "quality. Output nothing else."
)

conversation = [
    {"role": "system", "content": [{"type": "text", "text": system_prompt}]},
    {"role": "user", "content": [
        {"type": "audio", "audio": "audio.wav"},
        {"type": "text", "text": "Source: I love cake.\nTranslation: Ich liebe Kekse."}
    ]},
]

text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info([conversation], use_audio_in_video=False)
inputs = processor(
    text=text, audio=audios, images=images, videos=videos,
    return_tensors="pt", use_audio_in_video=False
).to(model.device).to(model.dtype)
input_len = inputs["input_ids"].shape[1]

with torch.no_grad():
    output_ids = model.generate(**inputs, use_audio_in_video=False, max_new_tokens=16, return_audio=False)

score = processor.decode(output_ids[0][input_len:], skip_special_tokens=True).strip()
print(float(score))

Citation

bibtex

@misc{züfle2026needspeechevaluatespeech,
      title={Why We Need Speech to Evaluate Speech Translation},
      author={Maike Züfle and Danni Liu and Vilém Zouhar and Jan Niehues},
      year={2026},
      eprint={2605.28227},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.28227},
}