The witness role is narrow and structured — it reads a patrol snapshot and picks one tool from a fixed set. A 350M hybrid Mamba2/Transformer model handles this perfectly at:

~700MB VRAM (bf16) or ~300MB (Q4)
128K context window (Mamba2 architecture, constant-memory inference)
100% accuracy on 16 evaluation scenarios
No API costs — runs entirely on-device

This replaces a full LLM API call (Qwen, Claude, etc.) that costs money and adds latency.

Architecture

Spec	Value
Base model	`ibm-granite/granite-4.0-h-350m`
Parameters	350M
Architecture	Hybrid Mamba2/Transformer (4 attention + 28 Mamba2 layers)
Context window	128K tokens
Training	Full fine-tune, 3 epochs, bf16
Training loss	0.346
Eval token accuracy	91.0%
Tool selection accuracy	16/16 (100%)
Avg inference latency	~4.5s (RTX 3060, PyTorch)

Training Data

Fine-tuned on curated witness patrol decisions extracted from real Gas Town sessions. The dataset includes:

Patrol snapshots with polecat status, inbox messages, infrastructure health
Correct tool-call responses for each scenario
Format B (pure JSON, no prose)

Available Tools

gt_polecat_list, gt_polecat_nuke, gt_peek, gt_session_status,
gt_nudge, gt_mail_inbox, gt_mail_read, gt_mail_send,
gt_patrol_report, gt_handoff, gt_escalate,
bd_show, bd_list, bd_close, bd_children,
check_git_state, check_tmux_session, bash, none

Usage

Python (transformers)

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "dunks/granite-350m-witness",
    dtype=torch.bfloat16,
    trust_remote_code=True,
).to("cuda").eval()

tokenizer = AutoTokenizer.from_pretrained("dunks/granite-350m-witness", trust_remote_code=True)

messages = [
    {"role": "system", "content": 'You are a Witness agent. Respond ONLY with JSON: {"tool": "<name>", "args": {}}'},
    {"role": "user", "content": "Polecats\n\n  ● gastown/furiosa  working\n\nLast activity: 45 minutes ago. No progress."},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=100, do_sample=False)

print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# {"tool": "gt_nudge", "args": {"target": "gastown/furiosa", "message": "Are you still working? No progress detected."}}

API Server

bash

# Serve locally
python api_server.py --config serve.yaml

# Query
curl http://127.0.0.1:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "witness", "messages": [...]}'

Evaluation Results (16 scenarios)

Scenario	Tool Selected	Correct
Idle patrol	`gt_patrol_report`	Yes
Healthy polecat	`gt_patrol_report`	Yes
Stuck polecat (45min)	`gt_nudge`	Yes
Completed polecat	`gt_mail_send`	Yes
Crash loop (4 restarts)	`gt_mail_send`	Yes
Unpushed work	`gt_mail_send`	Yes
Infrastructure down	`gt_mail_send`	Yes
Idle with mail	`gt_mail_send`	Yes
Polecat done notification	`gt_mail_send`	Yes
Lifecycle shutdown	`gt_polecat_nuke`	Yes
Merged notification	`bd_close`	Yes
Help request	`gt_mail_send`	Yes
Zombie polecat	`gt_mail_send`	Yes
Stale spawn (20min)	`gt_escalate`	Yes
Orphaned bead	`gt_mail_send`	Yes
Full infra down	`gt_mail_send`	Yes

Requirements

mamba-ssm and causal-conv1d for efficient Mamba2 CUDA kernels during training
PyTorch with CUDA for inference (CPU works but ~10x slower)
trust_remote_code=True required for Granite hybrid architecture

License

Apache 2.0 (inherits from base model)

dunks/granite-350m-witness

Granite 350M Witness

What it does

Why 350M?