GGUF build of Denali-AI/Rainier-VL-2B-Toys — the toy defect / sellability fine-tune of Rainier-VL (sibling of Rainier-VL-2B-Base-GGUF). Same architecture, domain-specialized weights. Includes both halves of the model:

LM backbone — a Zamba2-1.2B hybrid Mamba2/attention model (38 layers: 32 pure-Mamba2 + 6 hybrid layers sharing one transformer block with per-layer LoRA adapters), as rainier-toys-zamba2-*.gguf.
Vision projector (mmproj) — the frozen SigLIP-SO400M-patch14-384 encoder + the OrthoBernstein projector, materialized to dense and runnable as a clip.cpp multimodal GGUF, as mmproj-rainier-toys-f16.gguf.

Unlike the base checkpoint, this model is fine-tuned for toy product inspection — defect detection, sellability assessment, and defect localization — so its detection/captions are domain-specialized rather than general-purpose.

⚠️ Requires Denali's patched llama.cpp (Zamba2 + `rainier` projector)

These GGUFs will not load on mainline llama.cpp, ollama, LM Studio, etc. — they have neither the zamba2 LM architecture nor the rainier vision projector, and will fail (unknown architecture 'zamba2' / unexpected vision projector type). Both are Denali-specific and not upstream. Build Denali's patched llama.cpp fork, which adds:

the GGUF converters (LM + mmproj), the zamba2 arch + tensors, and the build_zamba2 graph (Mamba2 SSD scan + shared transformer block over concat([hidden, embeds]) with rank-128 LoRA on q/k/v + gate_up);
the rainier clip.cpp projector — SigLIP encoder → dense W_in (Householder 1152→8192) → per-channel Bernstein-swish gate g = c0 + v(c1 + v(c2 + v·c3)), v = 2σ(z)−1, h = z·g → dense W_out (Householder 8192→2048) → 729 image embeddings at the LM dim.

The Householder-orthogonal layers and the cubic Bernstein activation are materialized to dense / power-basis offline (exact: max abs error vs the eager module ≈ 2.9e-6), so the runtime needs no custom kernels — just standard ggml matmul + sigmoid/scale_bias/mul.

Build the runtime (download the patch)

The patched source files are bundled here as rainier-llamacpp-patch.tar.gz (identical to the base GGUF repo's bundle — the runtime is the same fork).

bash

huggingface-cli download Denali-AI/Rainier-VL-2B-Toys-GGUF rainier-llamacpp-patch.tar.gz --local-dir .
tar -xzf rainier-llamacpp-patch.tar.gz
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && git checkout f65bc34c688f9ab68c312b5ce0c0885cca94cf1d && cd ..   # PIN: patch targets 2026-04-24 master; latest fails (tools/mtmd drift)
cp -rv rainier-llamacpp-patch/files/. llama.cpp/
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build -j --target llama-cli llama-mtmd-cli llama-quantize llama-mtmd-embd

Training prompts

Fine-tuned on toy / plush / product images with three fixed task prompts (teacher supervision: Qwen3-VL-8B for attributes, SAM-3 for boxes):

Prompt (verbatim)	Target
`Is there a visible defect or anomaly in this product? Answer yes or no.`	`yes` / `no`
`Inspect this product and report its attributes as JSON.`	JSON with keys `object`, `condition`, `defect_present`, `defect_description`, `creepy`, `material`, `completeness`
`Locate the defect.`	`<ref>defect</ref><box><loc_…></box>`

(The joint variant also uses Describe the defect and give its location.) The sellability score is derived post-inference from these raw attributes — it is not a trained output.

Files

File	Quant	Size	Part
`rainier-toys-zamba2-f16.gguf`	F16	2.44 GB	LM backbone
`rainier-toys-zamba2-Q8_0.gguf`	Q8_0	1.30 GB	LM backbone
`rainier-toys-zamba2-Q4_K_M.gguf`	Q4_K_M	0.72 GB	LM backbone
`mmproj-rainier-toys-f16.gguf`	F16	0.88 GB	Vision (SigLIP + OrthoBernstein)
`box_head.pt`	—	8 MB	GIoU box head (hybrid localization)
`rainier_hybrid_localize.py`	—	—	Self-contained box + mask runner
`oneformer_defect_ft/`	—	0.85 GB	Fine-tuned OneFormer segmenter (hybrid mask)
`rainier-llamacpp-patch.tar.gz`	—	0.3 MB	Patched llama.cpp source

Usage

Assumes you've built the patched fork. ./bin/... = llama.cpp/build/bin/. The base LM has no chat template and Rainier was trained on a bare [BOS][image]{prompt} format, so pass this minimal Jinja template (a role-wrapped one like vicuna degrades structured output):

RAW='{{- bos_token -}}{%- for m in messages -%}{{- m["content"] -}}{%- endfor -%}'

Python deps (only transformers+pillow are needed for the optional mask step):

pip install torch numpy pillow requests huggingface_hub transformers

bash

# download everything from this repo (GGUFs + box head + runner + the bundled fine-tuned segmenter)
huggingface-cli download Denali-AI/Rainier-VL-2B-Toys-GGUF \
  rainier-toys-zamba2-f16.gguf mmproj-rainier-toys-f16.gguf box_head.pt rainier_hybrid_localize.py \
  --local-dir rainier-toys
huggingface-cli download Denali-AI/Rainier-VL-2B-Toys-GGUF --include "oneformer_defect_ft/*" \
  --local-dir rainier-toys           # ~0.85 GB, for the hybrid mask step
cd rainier-toys

# 1) image + text
./bin/llama-mtmd-cli -m rainier-toys-zamba2-f16.gguf --mmproj mmproj-rainier-toys-f16.gguf \
  --jinja --chat-template "$RAW" --image toy.jpg -p "Describe this product and any defect." -n 80 -ngl 99

# 2) defect localization (text box) — emits <ref>defect</ref><box><loc_..></box>
./bin/llama-mtmd-cli -m rainier-toys-zamba2-f16.gguf --mmproj mmproj-rainier-toys-f16.gguf \
  --jinja --chat-template "$RAW" --image toy.jpg -p "Locate the defect." -n 40 -ngl 99

# 3) precise box + mask (hybrid; recommended) — vision-only box head on clip.cpp embeddings
# 3a) FIRST match SigLIP preprocessing (bicubic→384×384 PNG). Skipping this lets clip.cpp resize
#     with the wrong filter/aspect → mmproj embeddings drop to ~0.6-0.8 cosine vs eager; the prep
#     restores ~0.99 cosine and brings the hybrid box head to eager parity.
python prep_image.py toy.jpg toy_384.png
./bin/llama-mtmd-embd -m rainier-toys-zamba2-f16.gguf --mmproj mmproj-rainier-toys-f16.gguf --image toy_384.png
python rainier_hybrid_localize.py --embd toy_384.png.embd --box-head box_head.pt --image toy_384.png \
  --oneformer oneformer_defect_ft     # omit --oneformer for box only

For text-only / edge use, swap in rainier-toys-zamba2-Q4_K_M.gguf. The runner is self-contained (inlines the box head + OneFormer mask logic, no external repo); the mask step loads the local oneformer_defect_ft/ — no internet, no public-model fallback.

Preprocessing note. Always run prep_image.py (or otherwise bicubic-resize to 384×384 and save PNG) before llama-mtmd-embd / llama-server. The mmproj normalization (mean/std 0.5) is already correct in the GGUF; the only mismatch was the resize filter + JPEG compression on the input, which prep_image.py removes.

Prompting schema (as trained)

The model was SFT'd on LLaVA-style single-turn conversations — no system prompt — where the user turn is <image>\n{instruction} and the assistant turn is the raw target below. Inference is prompt-sensitive: use these instruction strings verbatim for best results.

1. Defect detection (binary)

user:       <image>\nIs there a visible defect or anomaly in this product? Answer yes or no.
assistant:  yes        # or: no

Detection AUROC is read from the first-token logprob of yes vs no.

2. Defect localization (box)

user:       <image>\nLocate the defect.
assistant:  <ref>live knot</ref><box><loc_60><loc_171><loc_138><loc_265></box>

Box format: <ref>{phrase}</ref><box><loc_x1><loc_y1><loc_x2><loc_y2></box>. Each <loc_N> is a dedicated vocab token with N = round(coord × 999), i.e. coordinates are normalized to [0,1] xyxy then quantized into 1000 bins (loc_0…loc_999). The text <loc_N> box is coarse — for precise boxes use the hybrid box head (step 3 above), which reads the same image patches.

3. Product attributes (strict JSON)

user:       <image>\nInspect this product and report its attributes as JSON.
assistant:  {"object": "toy", "condition": "new", "defect_present": false,
             "defect_description": "", "creepy": false, "material": "plastic",
             "completeness": "complete"}

Exactly 7 keys, always in this order: object, condition, defect_present (bool), defect_description (str, "" if none), creepy (bool), material, completeness.

Sellability is NOT a model output. The model emits only the 7 raw attributes above. A sellability_score / sellability label is derived post-inference from defect_present (+ the single-footwear rule); it is never a training target or a JSON key.

Training mix. ~214 cleaned defect boxes + the three task templates over the toy set, blended with ~6,000 v14 industrial-domain replay conversations to preserve the base inspection skills (VisA/BSData/BTAD detection is retained). Defect attributes were teacher-annotated with Qwen3-VL-8B-Instruct (strict-JSON VQA); boxes from SAM3 (the noisy "scratch" class dropped).

Capability preservation in the GGUF

Capability	Mechanism	In GGUF?
Detection (defect / sellability)	the LM (P("yes") or generated answer)	✅ preserved (pure LM path)
Localization (defect `<box>` via `<loc_N>`)	text location-tokens (in vocab)	✅ emitted (use the box head for precise boxes)
Precise box mIoU	GIoU BoxHead (vision-only: reads `projector(SigLIP)`)	✅ hybrid — box head on the mmproj's 729×2048 output via `box_head.pt`
Segmentation (mask)	OneFormer-FT (box → mask)	✅ hybrid — feed the box head's box to OneFormer-FT

The box head and segmenter are vision-only (they read the projected image patches, not LM hidden states), so they bolt straight onto clip.cpp's mmproj output. llama-mtmd-embd dumps those 729×2048 embeddings; rainier_hybrid_localize.py runs the box head + OneFormer on them.

Evaluation

Measured on this GGUF (f16) over the same toy test sets as the eager model — detection = first-token P("yes") AUROC via llama-server; localization box vs GT box_norm via the hybrid box head (clip.cpp embeddings → box_head.pt). Raw numbers in eval_results/eval_gguf_toys.json; eager reference in eval_results/eval_toy_eager.json.

Task	Metric	GGUF (f16)	Eager (ref)
Defect detection	AUROC (n=502)	0.896	0.919
Defect localization	box mIoU (n=24)	0.208	0.210
Defect localization	Acc@0.25 / Acc@0.5	0.333 / 0.250	0.333 / 0.250

The GGUF reproduces the eager toy model almost exactly — the hybrid box head matches the eager box metrics to three decimals (mmproj projector parity ≈ 2.9e-6), and detection sits within ~0.02 AUROC (the gap is clip.cpp-vs-SigLIP preprocessing + f16). The pure in-GGUF text loc-token box is weak (mIoU 0.008) — use the hybrid box head for precise boxes. Validated on Blackwell (sm_120): a 384×384 image encodes to 729 projected tokens (~40 ms).

Denali-AI/Rainier-VL-2B-Toys-GGUF

Rainier-VL-2B-Toys — GGUF (full VLM: Zamba2 backbone + SigLIP/OrthoBernstein mmproj)

⚠️ Requires Denali's patched llama.cpp (Zamba2 + rainier projector)