grug bird keep same club skill. brain voice now actual grug.
July 15, 2026 default-brain audit: 33 fresh neutral prompts, including 15
tool-enabled coding-agent prompts and the three reported planner-English failure
shapes, gave reasoning on 33/33, Grug-clean reasoning on 33/33, and zero hidden
style instructions. This repo already used original Ornith chat template with no
Grug prompt, so grug not replace good weight merely to claim new weight. Tiny
depth-only candidate made no meaningful improvement and stayed rejected. Main
weights and strong coding/tool scores below remain same; this note records audit.
grug honest release note: old main-branch weights replaced after dialect repair.
same repo name, new merged checkpoint. pre-repair rock stays on backup branch
pre-dialect-fix-2026-07-13. local cache user should redownload.
old bird sometimes think: "User wants hello world Python. Simple code snippet,
no tools needed. Provide code and brief explanation." short, yes. grug, no.
new bird think:
Need Python hello-world. Tiny valid snippet enough. Then one-line explain.
dialect fix train only private <think> target.
human answer and tool call get no correction loss. joint gate require style
improve + every coding/tool score stay whole.
this full merged 9b model. no adapter needed.
whole bird comparison
Same greedy harness, prompts, parser, runtime, and limits. HumanEval 164 tasks;
MBPP first 100 sanitized test tasks; card 18 held-out actions; broad 119 held-out
actions.
model
HumanEval
MBPP
card valid
card strict
card right
broad valid
broad strict
broad right
Ornith 1.0 9B
86.6
76.0
72.2
61.1
61.1
100.0
92.4
62.2
Grug v1 9B
78.7
76.0
88.9
88.9
88.9
99.2
86.6
91.6
Grug v2 before dialect fix
81.1
77.0
100.0
100.0
100.0
100.0
100.0
92.4
Grug v2 corrected
82.9
77.0
100.0
100.0
100.0
100.0
100.0
94.1
same-runtime rerun matter. old card number from different vLLM build not mixed into
table. exact JSON rock included in results/.
grug family benchmark
Same prompts, parser, runtime, decoding, and limits for both birds. All numbers are
percent; bold marks the best result in each column. Ties make both rocks bold.
model
HumanEval
MBPP
card valid
card strict
card right
broad valid
broad strict
broad right
Grug v2 9B
82.9
77.0
100.0
100.0
100.0
100.0
100.0
94.1
Grug 35B
80.5
88.0
94.4
88.9
94.4
100.0
100.0
95.0
dialect-fix capability gate
test
before dialect fix
after
change
HumanEval pass@1 %
81.1
82.9
+1.8
MBPP pass@1 %
77.0
77.0
+0.0
card valid tool %
100.0
100.0
+0.0
card strict tool %
100.0
100.0
+0.0
card right tool %
100.0
100.0
+0.0
broad valid tool %
100.0
100.0
+0.0
broad strict tool %
100.0
100.0
+0.0
broad right tool %
92.4
94.1
+1.7
valid = parser find offered tool call. strict = exact schema + required args.
right tool = expected next action, not merely valid different club.
dialect gate
Separate 90-prompt held-out suite: 50 trivial, 20 moderate, 20 complex. No
prompt used for gradient.
measure
before
after
dialect-clean trace %
1.11
100.0
function-word ratio %
7.25
2.44
User asks/wants/... trace
89
0
no tools needed trace
70
0
need to trace
4
0
complex think median word
28
25
complex median gate protect brain meat: after must keep at least 80% old median
and at least 25 word. grug remove grammar, not reasoning branch.
