SpeechLLM is a multi-modal LLM trained to predict the metadata of the speaker's turn in a conversation. speechllm-2B model is based on HubertX audio encoder and TinyLlama LLM. The model predicts the following:
SpeechActivity : if the audio signal contains speech (True/False)
Transcript : ASR transcript of the audio
Gender of the speaker (Female/Male)
Age of the speaker (Young/Middle-Age/Senior)
Accent of the speaker (Africa/America/Celtic/Europe/Oceania/South-Asia/South-East-Asia)
Emotion of the speaker (Happy/Sad/Anger/Neutral/Frustrated)
Usage
python
1# Load model directly from huggingface2from transformers import AutoModel
3model = AutoModel.from_pretrained("skit-ai/speechllm-1.5B", trust_remote_code=True)45model.generate_meta(6 audio_path="path-to-audio.wav",#16k Hz, mono7 audio_tensor=torchaudio.load("path-to-audio.wav")[1],# [Optional] either audio_path or audio_tensor directly8 instruction="Give me the following information about the audio [SpeechActivity, Transcript, Gender, Emotion, Age, Accent]",9 max_new_tokens=500,10 return_special_tokens=False11)1213# Model Generation14'''
15{
16 "SpeechActivity" : "True",
17 "Transcript": "Yes, I got it. I'll make the payment now.",
18 "Gender": "Female",
19 "Emotion": "Neutral",
20 "Age": "Young",
21 "Accent" : "America",
22}
23'''
Try the model in Google Colab Notebook. Also, check out our blog on SpeechLLM for end-to-end conversational agents(User Speech -> Response).
