Metric	Value
Size on disk	26.3 GB
Average bits per weight	6.01
Base type	Q4_K_M (per-tensor overrides via spec)
Framework	llama.cpp (GGUF)
Calibration	Importance matrix (wikitext-2 + MMLU-Pro, seed=99)
Source	Qwen/Qwen3.6-35B-A3B (BF16, 71.9 GB)

Bit distribution (620 quantized tensors)

Type	Count	Purpose
F16	363	Attention (q/k/v/o), MoE routers, SSM projections, norms
Q8_0	93	High-sensitivity MoE experts + shared-expert projections
Q6_K	63	Medium-sensitivity MoE experts
Q5_K	77	Lower-sensitivity MoE experts
Q4_K	7	Low-sensitivity SSM auxiliary tensors
Q3_K	17	SSM conv/gate tensors (lowest sensitivity)

Benchmarks

Measured on MMLU-Pro (140 stratified questions, 10 per subject, seed=42, thinking mode on):

Model	MMLU-Pro Accuracy	Size
MLX RAM sibling	77.9%	25.9 GB
F16 GGUF (unquantized ceiling)	70.7%	65 GB
This model	69.3%	26.3 GB

Captures >97% of the GGUF format's achievable quality at ~40% of the F16 size. The gap between MLX and F16 GGUF reflects llama.cpp's handling of Qwen3.6's hybrid linear-attention + full-attention + SSM architecture, not quantization overhead.

Usage

Every option below is a complete start-from-zero recipe. Pick one.

Disk space: leave ~30 GB free. The GGUF is 26.3 GB; some runtimes keep a working copy.

Option 1 — llama.cpp (CLI)

bash

# 1. Install llama.cpp (macOS / Linux)
brew install llama.cpp huggingface-cli
# On Linux without brew:
#   pip install huggingface_hub
#   see https://github.com/ggml-org/llama.cpp for build instructions

# 2. Download this model (~26 GB)
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir ./qwen3.6-ram-26gb

# 3. Run interactively
llama-cli -m ./qwen3.6-ram-26gb/Qwen3.6-35B-A3B-RAM-26GB.gguf \
    -p "Explain quantum entanglement in one paragraph." \
    -n 512 --threads 8 -ngl 99

Option 2 — llama-server (OpenAI-compatible HTTP API)

bash

# 1-2. Install + download (same as Option 1)
brew install llama.cpp huggingface-cli
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir ./qwen3.6-ram-26gb

# 3. Start the server
llama-server \
    -m ./qwen3.6-ram-26gb/Qwen3.6-35B-A3B-RAM-26GB.gguf \
    --port 8080 -ngl 99 --ctx-size 8192

# 4. Query it (in another terminal)
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [{"role": "user", "content": "Hello!"}],
        "max_tokens": 256,
        "temperature": 1.0
    }'

Option 3 — Ollama

Assumes Ollama is already installed and its daemon is running.

bash

# 1. Download the GGUF into a working directory
mkdir -p qwen3.6-ram-26gb && cd qwen3.6-ram-26gb
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir .

# 2. Create a Modelfile next to the GGUF
cat > Modelfile <<'EOF'
FROM ./Qwen3.6-35B-A3B-RAM-26GB.gguf
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 8192
EOF

# 3. Import into Ollama and run
ollama create qwen3.6-ram-26gb -f Modelfile
ollama run qwen3.6-ram-26gb "Hello!"

Option 4 — LM Studio (GUI)

Install LM Studio and launch it
Search for baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF in the Discover tab (or paste the repo URL)
Click Download — LM Studio will fetch the GGUF into its local cache
Load the model from the My Models tab; use the Chat tab to converse

Option 5 — Python (`llama-cpp-python`)

bash

# 1. Install bindings
pip install llama-cpp-python huggingface_hub

# 2. Download the GGUF
hf download baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF \
    --include "*.gguf" \
    --local-dir ./qwen3.6-ram-26gb

# 3. Use it
python - <<'PY'
from llama_cpp import Llama
llm = Llama(
    model_path="./qwen3.6-ram-26gb/Qwen3.6-35B-A3B-RAM-26GB.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,   # all layers to GPU on Apple Silicon / CUDA
)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=256,
    temperature=1.0,
)
print(out["choices"][0]["message"]["content"])
PY

Recommended inference settings

temperature: 1.0
top_p: 0.95
top_k: 40
repetition_penalty: 1.1
max_tokens: 8192

Enable thinking mode in the chat template for best accuracy on reasoning tasks.

Quantization method

RAM probe allocator measures per-tensor sensitivity to quantization across bits 2–8
Structural-bounds + proxy prediction locates the global quality peak budget
Path B knapsack solver re-optimizes allocations in GGUF type space using GGUF-adjusted effective bits and actual GGUF byte costs
F16 kept for the most sensitive tensors (attention projections, MoE routers, SSM projections, norms)
Importance matrix computed from 100 chunks of wikitext-2 + 200 MMLU-Pro questions (seed=99, disjoint from evaluation seed)
llama-quantize applies the per-tensor spec with --imatrix

No fine-tuning, no calibration-dependent optimization — the probe runs on random Gaussian inputs only.

Quantized by baa.ai

Black Sheep AI Products

Shepherd — Private AI deployment platform that shrinks frontier models by 50-60% through RAM compression, enabling enterprises to run sophisticated AI on single GPU instances or Apple Silicon hardware. Deploy in your VPC with zero data leaving your infrastructure. Includes CI/CD pipeline integration, fleet deployment across Apple Silicon clusters, air-gapped and sovereign deployment support, and multi-format export (MLX, GGUF). Annual cloud costs from ~$2,700 — or run on a Mac Studio for electricity only.

Watchman — Capability audit and governance platform for compressed AI models. Know exactly what your quantized model can do before it goes live. Watchman predicts which capabilities survive compression in minutes — replacing weeks of benchmarking. Includes compliance-ready reporting for regulated industries, quality valley warnings for counterproductive memory allocations, instant regression diagnosis tracing issues to specific tensors, and 22 adversarial security probes scanning for injection, leakage, hallucination, and code vulnerabilities.

Learn more at baa.ai — Sovereign AI.

baa-ai/Qwen3.6-35B-A3B-RAM-26GB-GGUF

Qwen3.6-35B-A3B — 26GB (GGUF)

Files

Metrics

Bit distribution (620 quantized tensors)

Benchmarks

Usage

Option 1 — llama.cpp (CLI)

Option 2 — llama-server (OpenAI-compatible HTTP API)

Option 3 — Ollama

Option 4 — LM Studio (GUI)

Option 5 — Python (llama-cpp-python)

Recommended inference settings

Quantization method

Black Sheep AI Products

baa-ai/
Qwen3.6-35B-A3B-RAM-26GB-GGUF

Option 5 — Python (`llama-cpp-python`)