You can deploy this model and start earning money today!

Menlo/
llama3.1-s-instruct-2024-08-15-cp2000

Name: llama3.1-s-instruct-2024-08-15-cp2000
Availability: InStock

pytorch

llama

sound language model

homebrewltd/instruction-speech-whispervq-v2

apache-2.0

Views

No views yet

Caution

This is an intermediate checkpoint.

Model Details

We have developed and released the family llama3s. This family is natively understanding audio and text input.

We continue to supervised finetune our last checkpoint using WhisperVQ as a tokenizer for audio files homebrewltd/... with 2B tokens from Instruction Speech WhisperVQ v2 dataset.

Model developers Homebrew Research.

Input Text and sound.

Output Text.

Model Architecture Llama-3.

Language(s): English.

Intended Use

Intended Use Cases This family is primarily intended for research applications. This version aims to further improve the LLM on sound understanding capabilities.

Out-of-scope The use of llama3-s in any manner that violates applicable laws or regulations is strictly prohibited.

How to Get Started with the Model

First, we need to convert the audio file to sound tokens

Then, we can inference the model the same as any other LLM.

Training process

Training Metrics Image: Below is a snapshot of the training loss curve visualized.

Hardware

GPU Configuration: Cluster of 8x NVIDIA H100-SXM-80GB. GPU Usage:

Continual Training: 6 hours.

Training Arguments

We utilize torchtune library for the latest FSDP2 training code implementation.

Parameter	Continual Training
Epoch	1
Global batch size	128
Learning Rate	0.5e-4
Learning Scheduler	Cosine with warmup
Optimizer	Adam torch fused
Warmup Ratio	0.01
Weight Decay	0.005
Max Sequence Length	1024

Citation Information

BibTeX:

@article{Llama3-S: Sound Instruction Language Model 2024,
  title={Llama3-S},
  author={Homebrew Research},
  year=2024,
  month=August},
  url={https://huggingface.co/homebrewltd/llama3.1-s-2024-08-15}

Acknowledgement

WhisperSpeech
Meta-Llama-3.1-8B-Instruct

Menlo/llama3.1-s-instruct-2024-08-15-cp2000