You can deploy this model and start earning money today!

yejingfu/
Meta-Llama-3.1-8B-Instruct-FP8-128K

Name: Meta-Llama-3.1-8B-Instruct-FP8-128K
Availability: InStock

safetensors

llama

facebook

Meta-Llama3.1-8B-FP8-128K

Model Overview

Model Architecture: Meta-Llama-3.1
- Input: Text
- Output: Text
Model Optimizations:
- Weight quantization: FP8
- Activation quantization: FP8
- KV Cache quantization:FP8
Intended Use Cases: Intended for commercial and research use in multiple languages. Similarly to Meta-Llama-3.1-8B-Instruct, this models is intended for assistant-like chat.
Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
Release Date: 8/27/2024
Version: 1.0
License(s): llama3.1
Quantized version of Meta-Llama-3.1-8B-Instruct.

Serve with vLLM engine

bash

python3 -m vllm.entrypoints.openai.api_server \
    --port <port> --model yejingfu/Meta-Llama-3.1-8B-Instruct-FP8-128K \
    --tensor-parallel-size 1 --swap-space 16 --gpu-memory-utilization 0.96 \
    --max-num-seqs 32 --max-model-len 131072 --kv-cache-dtype fp8 --enable-chunked-prefill

yejingfu/Meta-Llama-3.1-8B-Instruct-FP8-128K

Meta-Llama3.1-8B-FP8-128K

Model Overview

Serve with vLLM engine

license: llama3.1

yejingfu/
Meta-Llama-3.1-8B-Instruct-FP8-128K