Here are some examples of using this model in MLC LLM. Before running the examples, please install MLC LLM by following the installation documentation.

Chat

In command line, run

mlc_llm chat HF://SeanChang74/Qwen2.5-0.5B-Instruct-q4f16_1-MLC

REST Server

In command line, run

mlc_llm serve HF://SeanChang74/Qwen2.5-0.5B-Instruct-q4f16_1-MLC

Python API

python

from mlc_llm import MLCEngine

# Create engine
model = "HF://SeanChang74/Qwen2.5-0.5B-Instruct-q4f16_1-MLC"
engine = MLCEngine(model)

# Run chat completion in OpenAI API.
for response in engine.chat.completions.create(
    messages=[{"role": "user", "content": "What is the meaning of life?"}],
    model=model,
    stream=True,
):
    for choice in response.choices:
        print(choice.delta.content, end="", flush=True)
print("\n")

engine.terminate()

Documentation

For more information on MLC LLM project, please visit our documentation and GitHub repo.