from llama_cpp import Llama

llm = Llama(model_path="model.gguf", chat_format="chatml", n_ctx=512, n_threads=4, verbose=False)
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=50,
)
print(response["choices"][0]["message"]["content"])

dispatchAI SDK

python

from dispatchai import load_model
model = load_model("SmolLM2-1.7B-Instruct-Q5-mobile", backend="gguf")
print(model.chat("What is the capital of France?"))

On Android (via ADB)

bash

hf download dispatchAI/SmolLM2-1.7B-Instruct-Q5-mobile model.gguf
MSYS_NO_PATHCONV=1 adb push model.gguf /data/local/tmp/
MSYS_NO_PATHCONV=1 adb shell "cd /data/local/tmp && LD_LIBRARY_PATH=/data/local/tmp ./llama-cli -m model.gguf -p 'Hello' -n 30 -t 4 -st"

Model Details

Attribute	Value
Base Model	unknown
File Size	1169 MB
Format	GGUF
Chat Format	chatml
License	apache-2.0

About dispatchAI

dispatchAI — Small. Mobile. Free. UAE-built.