Use temperature 0.2 and load system_prompt.txt as the system message. OpenAI-compatible API: http://127.0.0.1:8080/v1/chat/completions.
Inference defaults are in config.recommended.json.
Evaluation Results
Measured locally on NVIDIA RTX 3060 12GB with lm-evaluation-harness v0.4.12 and llama-server (20 examples per task). Frontier scores are vendor-published references (see frontier_reference.json).
Evaluated: 2026-07-15
Overall Comparison
Benchmark comparison
Benchmark
What it tests
Meltdown Q8 (3B)
GPT-4.1
Claude Sonnet 4
Gemini 2.5 Pro
GSM8K (8-shot, n=20)
Grade-school math word problems
65.0%
95.2%
94.0%
93.5%
MMLU HS Mathematics (n=20)
High school math (generative MC)
0.0%
90.2%
88.5%
89.0%
MMLU HS Computer Science (n=20)
HS CS knowledge (generative MC)
0.0%
90.2%
88.5%
89.0%
IFEval (strict, n=20)
Instruction following
60.0%
87.5%
86.0%
85.5%
Agent Eval (n=20)
Structured tool-use on held-out prompts
60.0%
—
—
—
Raw JSON: eval_standard.json, eval_agent.json.
While the benchmark scores themselves are low, it is due to image-related benchmarks. For chat, the APE format overrides the base chat template, which is why it scores low in chat.
In day to day agentic workflows, with abbreviated MCP tools and unified instructions, Meltdown performs exceptionally. Escpecially while utilizing local Agentic harnesses like iRun's ReAct and APE.
Agent Eval Breakdown
Agent eval by category
Split
Pass Rate
Tests
Tool & file tasks
80.0%
12/15
Conversational tasks
0.0%
0/5
Overall
60.0%
12/20
Local vs Cloud
Local vs cloud
Meltdown Q8
GPT-4.1
Claude Sonnet 4
Gemini 2.5 Pro
Runs offline
Yes
No
No
No
API key required
No
Yes
Yes
Yes
Data leaves your machine
No
Yes
Yes
Yes
Parameters
3B
—
—
—
Meltdown is optimized for local agent work (privacy, zero API cost, offline use), not for beating frontier models on broad knowledge benchmarks.
