I'm a PhD student in visual neuroscience at the University of Toronto who also happens to spend way too much time fine-tuning, merging, and quantizing open-weight models on rented H100s and a local DGX Spark. All training compute is self-funded — balancing GPU costs against a student budget. If my uploads have been useful to you, consider buying a PhD student a coffee. It goes a long way toward keeping these experiments running.

Support on Ko-fi

Model info

Architecture: Qwen3_5MoeForCausalLM (linear + full attention interleaved, Gated Delta Net)
Parameters: 34.66 B total / ~3 B active (256 experts, 8 active per token)
Context: 262,144 tokens
Hidden size / layers: 2048 / 40
Vocab: 248,320 tokens

All sub-8-bit quants were produced with an importance matrix (imatrix) computed from a mixed-domain multilingual calibration corpus (eaddario/imatrix-calibration → combined_all_medium): 200 chunks × 512 tokens = 102,400 calibration tokens.

Quant index

Choose a quant that fits in your RAM/VRAM with room for context. For MoE models quality degrades more sharply at low bit widths than for dense models of similar size — prefer Q4_K_M or higher if you have the memory.

File	Bits	Size	imatrix	Notes
`Ornstein3.6-35B-A3B-Q8_0.gguf`	8	36.9 GB	—	Reference, near-lossless
`Ornstein3.6-35B-A3B-Q6_K.gguf`	6.5	28.5 GB	—	Great default for 32 GB+ systems
`Ornstein3.6-35B-A3B-Q5_K_M.gguf`	5.5	24.7 GB	✓	Excellent quality/size balance
`Ornstein3.6-35B-A3B-Q5_K_S.gguf`	5.5	24.0 GB	✓	Slightly smaller Q5
`Ornstein3.6-35B-A3B-Q4_K_M.gguf`	4.5	21.2 GB	✓	Common 24 GB-card default
`Ornstein3.6-35B-A3B-Q4_K_S.gguf`	4.5	19.9 GB	✓	Smaller Q4
`Ornstein3.6-35B-A3B-IQ4_XS.gguf`	4.25	~18 GB	✓	Smaller than Q4_K_S, comparable quality with imatrix
`Ornstein3.6-35B-A3B-Q3_K_M.gguf`	3.5	16.8 GB	✓	Usable; quality below Q4
`Ornstein3.6-35B-A3B-Q3_K_S.gguf`	3.5	15.2 GB	✓	Smaller Q3
`Ornstein3.6-35B-A3B-IQ3_M.gguf`	3.3	~15 GB	✓	Mixed I-quant, beats Q3_K_S at similar size
`Ornstein3.6-35B-A3B-IQ3_XXS.gguf`	3.0	~13 GB	✓	Aggressive 3-bit
`Ornstein3.6-35B-A3B-Q2_K.gguf`	2.6	12.9 GB	✓	Lowest K-quant; expect degraded quality
`Ornstein3.6-35B-A3B-IQ2_M.gguf`	2.7	~12 GB	✓	Aggressive I-quant 2-bit
`imatrix.dat`	—	192 MB	—	Importance matrix (GGUF format)

Usage

llama.cpp

bash

# Interactive chat
llama-cli -m Ornstein3.6-35B-A3B-Q4_K_M.gguf -cnv

# Single prompt
llama-cli -m Ornstein3.6-35B-A3B-Q5_K_M.gguf -p "Write a haiku about MoE routing."

# OpenAI-compatible server
llama-server -m Ornstein3.6-35B-A3B-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -c 8192

Other runners

LM Studio, Ollama (via a Modelfile), koboldcpp, and text-generation-webui all load these GGUFs provided their bundled llama.cpp supports Qwen3_5MoeForCausalLM with Gated Delta Net.

Reproducing the quants

bash

# 1. Convert safetensors → BF16 GGUF
python llama.cpp/convert_hf_to_gguf.py <model_dir> \
    --outtype bf16 --outfile Ornstein3.6-35B-A3B-BF16.gguf

# 2. Importance matrix
llama-imatrix \
    -m Ornstein3.6-35B-A3B-BF16.gguf \
    -f calibration.txt \
    -o imatrix.dat \
    --chunks 200 -c 512 -b 512 -ngl 99

# 3. Quantize (example)
llama-quantize --imatrix imatrix.dat \
    Ornstein3.6-35B-A3B-BF16.gguf \
    Ornstein3.6-35B-A3B-Q4_K_M.gguf Q4_K_M

License

Apache 2.0 — inherited from the Qwen 3.6 base release.

Nishant2414/Ornstein3.6-35B-A3B-GGUF

Ornstein3.6-35B-A3B-GGUF

Support This Work