thread13/

Qwen3.5-0.8B-llamafile

About

• Update(1) 2026-04-24: apparently, VirusTotal does not like the llamafiles and marks them "suspicious". There's not much I can do there, I guess. Let us wait untill the other scanners try it; until then -- feel free to e.g. ClamAV scan them after download, but also feel free to redo it yourself following the links provided: your result will be binary compatible with what is uploaded here.
• Update(2) 2026-04-24: hmm, the "suspicious" flag seems to be cleared on some binaries. Good. May be just give it time to update the databases.
• Update(3) 2026-04-24: files in this repository are marked as "safe" at the moment of writing this (2026-04-24).

• "and/or" here has a literal meaning -- one can either run a console chat (--chat), or a web server (--server), or -- by default -- both.
• NB(1): by default the web server attempts to listen on all available interfaces -- run it as --host 127.0.0.1 to make it private to your system only!
• NB(2): there's also a cli completion interface (--cli), if you wish -- check the docs!

How to use:

• https://github.com/mozilla-ai/llamafile/blob/main/docs/running_llamafile.md
• https://github.com/mozilla-ai/llamafile/blob/main/docs/quickstart.md

Building references:

• https://github.com/mozilla-ai/llamafile/blob/main/docs/creating_llamafiles.md
• https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF/blob/main/mmproj-BF16.gguf
• https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF/blob/main/Qwen3.5-0.8B-UD-Q8_K_XL.gguf

Default arguments:

• https://unsloth.ai/docs/models/qwen3.5

temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

// recommended (unsloth) for coding in thinking mode:
--ctx-size 16384 --temp 0.6 --top-p 0.95  --top-k 20  --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":true}'

// recommended (unsloth) for reasoning tasks:
--ctx-size 16384 --temp 1.0  --top-p 0.95 --top-k 20  --min-p 0.00 \

To enable "thinking mode" (slower but better results):

• https://unsloth.ai/docs/models/qwen3.5#how-to-enable-or-disable-reasoning-and-thinking

Extra recommendations -- try the following if generation falls into an endless loop:

• https://unsloth.ai/docs/models/qwen3.5 (same source)

Quirks:

• https://github.com/mozilla-ai/llamafile/issues/373
• https://github.com/mozilla-ai/llamafile/tree/0.9.0?tab=readme-ov-file#gotchas-and-troubleshooting