You can deploy this model and start earning money today!

mlc-ai/
Llama-3.1-70B-Instruct-fp8-MLC

Name: Llama-3.1-70B-Instruct-fp8-MLC
Availability: InStock

mlc-llm

meta-llama/Llama-3.1-70B-Instruct

finetune

Views

No views yet

Llama-3.1-70B-Instruct-fp8-MLC

This is the Llama-3.1-70B-Instruct model in MLC format e4m3_e4m3_f16 (FP8 quantization). The model can be used for projects MLC-LLM.

Example Usage

Here are some examples of using this model in MLC LLM. Before running the examples, please install MLC LLM by following the installation documentation.

Chat

In command line, run

mlc_llm chat HF://mlc-ai/Llama-3.1-70B-Instruct-fp8-MLC

REST Server

In command line, run

mlc_llm serve HF://mlc-ai/Llama-3.1-70B-Instruct-fp8-MLC

Python API

python

from mlc_llm import MLCEngine

# Create engine
model = "HF://mlc-ai/Llama-3.1-70B-Instruct-fp8-MLC"
engine = MLCEngine(model)

# Run chat completion in OpenAI API.
for response in engine.chat.completions.create(
    messages=[{"role": "user", "content": "What is the meaning of life?"}],
    model=model,
    stream=True,
):
    for choice in response.choices:
        print(choice.delta.content, end="", flush=True)
print("\n")

engine.terminate()

Documentation

For more information on MLC LLM project, please visit our documentation and GitHub repo.