You can deploy this model and start earning money today!

b4rtaz/
Llama-3_1-405B-Q40-Instruct-Distributed-Llama

Name: Llama-3_1-405B-Q40-Instruct-Distributed-Llama
Availability: InStock

distributed-llama

distributed-inference

text-generation

llama3.1

Views

No views yet

This is converted Llama 3.1 405B Instruct model to Distributed Llama format. The model is quantized to Q40 format. Due to Huggingface limitations, the model is split into 56 parts. Before use, you need to combine the parts together.

To run this model, you need approximately 240 GB of RAM on a single device, or approximately 240 GB of RAM distributed across 2, 4, 8, or 16 devices connected in a cluster (more informations how to do it you can find here).

🚀 How to Run?

⏬ Download the model. You have two options:

Download this repository and combine all parts together by using the cat command.
Download the model by using the launch.py script from Distributed Llama repository: python launch.py llama3_1_405b_instruct_q40

⏬ Download Distributed Llama repository.
🔨 Build Distributed Llama:

make dllama

🚀 Run Distributed Llama:

./dllama chat --model dllama_model_llama31_405b_q40.m --tokenizer dllama_tokenizer_llama_3_1.t --buffer-float-type q80 --max-seq-len 2048 --nthreads 64

🎩 License

You need to accept the Llama 3.1 license before downloading this model.

Llama 3.1 Community License

b4rtaz/Llama-3_1-405B-Q40-Instruct-Distributed-Llama

🚀 How to Run?

🎩 License

b4rtaz/
Llama-3_1-405B-Q40-Instruct-Distributed-Llama