This is the
Meta Llama 3.1 Instruct model fine‑tuned with LoRA and converted to MLC format
q4f16_1.
Before running any examples, install MLC LLM by following the
installation documentation.
1from mlc_llm import MLCEngine
2
3model = "HF://pr0methium/ReelevateLM-q4f16_1"
4engine = MLCEngine(model)
5
6for response in engine.chat.completions.create(
7 messages=[{"role": "user", "content": "Write me a 30 second reel story…"}],
8 model=model,
9 stream=True,
10):
11 for choice in response.choices:
12 print(choice.delta.content, end="", flush=True)
13print()
14
15engine.terminate()
For more information on the MLC LLM project, please visit the
docs and the
GitHub repo.
