GGUF quantizations of GestaltLabs/Ornstein-3.5-9B-V1.5 — a reasoning-focused fine-tune of Qwen 3.5 9B for AI-research and technical problem-solving, and an intermediate (V1.5) release between V1 and the upcoming V2 (a much more rigorous post-training run involving reinforcement-learning methods).

These builds keep the model's native multi-token-prediction (MTP) head (for speculative decoding in compatible runtimes) and ship a separate vision projector (mmproj) so the multimodal tower can be used for image/video input.

Files

File	Bits	Size	Notes
`Ornstein-3.5-9B-V1.5-Q8_0.gguf`	8	9.8 GB	near-lossless
`Ornstein-3.5-9B-V1.5-Q6_K.gguf`	6	7.6 GB	very high quality
`Ornstein-3.5-9B-V1.5-Q5_K_M.gguf`	5	6.6 GB	high quality
`Ornstein-3.5-9B-V1.5-Q4_K_M.gguf`	4	5.8 GB	recommended default
`Ornstein-3.5-9B-V1.5-Q3_K_M.gguf`	3	4.7 GB	smallest
`mmproj-Ornstein-3.5-9B-V1.5-f16.gguf`	f16	0.9 GB	vision encoder — pair with any quant for image/video

Usage (llama.cpp)

Text:

llama-cli -hf GestaltLabs/Ornstein-3.5-9B-V1.5-GGUF:Q4_K_M -p "Derive the variance of a sum of two correlated random variables."

Multimodal (image/video) — add the vision projector:

bash

llama-mtmd-cli -m Ornstein-3.5-9B-V1.5-Q4_K_M.gguf \
  --mmproj mmproj-Ornstein-3.5-9B-V1.5-f16.gguf \
  --image picture.jpg -p "Describe this image."

Tool calling (Hermes/Qwen format — the tool-capable chat template is embedded):

bash

llama-server -hf GestaltLabs/Ornstein-3.5-9B-V1.5-GGUF:Q4_K_M --jinja
# tools are passed in the OpenAI-compatible /v1/chat/completions "tools" field

Quality and speed scale with quant size; Q4_K_M is a strong default for ~8 GB of VRAM/RAM. The MTP head enables speculative decoding where supported.

Highlights (full-precision model)

Strong gains in multi-step reasoning over the base, with the largest improvements on hard, graduate-level scientific reasoning.
General conversational ability preserved.

Support This Work

I'm a PhD student in visual neuroscience at the University of Toronto who also happens to spend way too much time fine-tuning, merging, and quantizing open-weight models on rented H100s and a local DGX Spark. All training compute is self-funded — balancing GPU costs against a student budget. If my uploads have been useful to you, consider buying a PhD student a coffee. It goes a long way toward keeping these experiments running.

Support on Ko-fi

License

Apache 2.0 — inherited from the Qwen 3.5 9B base release.