Budgie-500M
Budgie-500M is a 487.8M-parameter English causal language model for compact mathematical reasoning, formal logic, science multiple choice, and dialogue with a 4,096-token context.
This root release promotes the strongest retained Budgie Alignment v2 checkpoint, verified-math-a025. The repository root contains inference artifacts only: no adapters, optimizer states, training rows, or obsolete checkpoints.
Full-split validation
The promoted weights were re-evaluated against the previous public Budgie on complete reasoning splits with identical user-only prompts and greedy decoding. This is a complete-split controlled evaluation under the documented scorer, not an official leaderboard submission.
| Benchmark | n | Budgie-500M | Previous public Budgie | Delta |
|---|
| GSM8K test | 1,319 | 18.57% (245/1,319) | 14.48% (191/1,319) | +4.09 pp |
| MATH-500 test | 500 | 12.00% (60/500) | 12.40% (62/500) | -0.40 pp |
| ARC-Challenge test | 1,172 | 27.56% (323/1,172) | 24.83% (291/1,172) | +2.73 pp |
| FOLIO validation | 203 | 35.47% (72/203) | 31.03% (63/203) | +4.43 pp |
| HelpSteer2 non-tied pairs | 373 | 45.31% (169/373) | 44.50% (166/373) | +0.80 pp |
| Four-task reasoning macro | — | 23.40% | 20.69% | +2.71 pp |
| Five-metric composite | — | 27.78% | 25.45% | +2.33 pp |
A 30,000-sample paired macro bootstrap gives a 95% composite-delta interval of [+0.68, +3.97] points and a 99.79% probability of a positive overall delta. GSM8K is the strongest individually confirmed gain. Full MATH-500 is a small, statistically insignificant regression (60 vs 62; paired p=0.888), so this release does not claim a MATH improvement.
Batching warning: ordinary padded batch generation is unsafe for this custom model. For exact behavior, generate sequentially or batch only prompts with identical tokenized input length, use no padding, and truncate each row at its first EOS.
Controlled comparison
All models used identical fixed rows and prompts with greedy decoding. Qwen used its native chat template with thinking disabled; Budgie used its native user/assistant template. This is a controlled regression suite, not a full official leaderboard run.
| Benchmark | n | Budgie-500M | Previous public Budgie | Qwen3-0.6B non-thinking |
|---|
| GSM8K | 30 | 16.67% (5/30) | 13.33% (4/30) | 46.67% (14/30) |
| MATH-500 | 15 | 20.00% (3/15) | 13.33% (2/15) | 13.33% (2/15) |
| ARC-Challenge | 30 | 36.67% (11/30) | 26.67% (8/30) | 63.33% (19/30) |
| FOLIO | 30 | 43.33% (13/30) | 36.67% (11/30) | 43.33% (13/30) |
| HelpSteer2 pairwise | 200 | 48.50% (97/200) | 48.00% (96/200) | 50.00% (100/200) |
| Unweighted composite | 5 metrics | 33.03% | 27.60% | 43.33% |
The controlled composite increases by 5.43 points over the previous public root.
Exact rows, revisions, prompts, token budgets, raw counts, and disclosures are in
benchmark.json.
Expanded disjoint check
A larger second gate excludes every row in the controlled table above.
| Benchmark | n | Budgie-500M | Previous public Budgie |
|---|
| GSM8K | 80 | 30.00% (24/80) | 11.25% (9/80) |
| MATH-500 | 50 | 20.00% (10/50) | 20.00% (10/50) |
| ARC-Challenge | 80 | 31.25% (25/80) | 20.00% (16/80) |
| FOLIO | 100 | 36.00% (36/100) | 31.00% (31/100) |
| HelpSteer2 pairwise | 373 | 45.31% (169/373) | 44.50% (166/373) |
| Unweighted composite | 5 metrics | 32.51% | 25.35% |
Paired bootstrap with 30,000 resamples estimates a +7.16 point composite delta, a 95% interval of [+2.59, +11.73] points, and a 99.86% bootstrap probability of a positive delta.
MATH-500 is tied on the expanded gate; the confirmed larger-sample gains are GSM8K, ARC-Challenge, FOLIO, and a small HelpSteer2 gain.
Dialogue regression check
A separate 36-prompt English dialogue suite covered explanations, rewriting, practical advice, uncertainty, natural conversation, and exact output constraints. Responses were judged twice with anonymized A/B order reversal. Qwen3.8 generated one side and also served as judge, so the comparison may contain self-model bias; objective repetition and constraint checks are included alongside it.
| Comparison | Budgie wins | Other wins | Ties |
|---|
| Current Budgie vs previous public Budgie | 8 | 6 | 22 |
| Current Budgie vs Qwen3.8-27B | 0 | 35 | 1 |
| Objective check | Current Budgie | Previous Budgie | Qwen3.8-27B |
|---|
| Repetition-loop flags (lower is better) | 10/36 | 20/36 | 1/36 |
| Exact constraint prompts passed | 1/6 | 2/6 | 6/6 |
| Honesty/uncertainty anchors passed | 6/6 | 2/6 | 6/6 |
The current checkpoint does not show an overall dialogue regression versus the previous public Budgie: it has a small net pairwise advantage, much stronger honesty, and half as many repetition flags. However, it remains far behind Qwen3.8 in factual explanations, writing, practical assistance, natural conversation, and constraint following. Raw outputs and order-swapped judgments are preserved in the Alignment v2 reports.
Post-training lineage
The release descends from the previous public Budgie checkpoint at revision 39c877f07ddc80aa975809a4d87c848372316caf.
The retained lineage combined controlled full-parameter SFT, Qwen3.8-27B distillation, high-confidence dual-teacher and preference replay, verified mathematics, and task-vector interpolation to limit forgetting. The final checkpoint applies 25% of the retained verified-math continuation to the preceding precision checkpoint.
No reported fixed test or validation row was used for training. The lineage predates the project's stricter benchmark-evaluation-only policy and did use training splits from GSM8K, ARC-Challenge, Hendrycks MATH, and ProofWriter, together with NuminaMath, Bespoke-Stratos, SmolTalk, Qwen3.8 synthetic data, and random verified mathematics. This is disclosed to prevent misleading benchmark claims.
Full experiment history, rejected runs, source policy, confidence reports, and handoff notes are preserved at:
Architecture
- 487,800,064 unique parameters
- 24 layers, hidden size 1,280
- 10 query heads, 2 KV heads, head dimension 128
- SwiGLU intermediate size 3,584
- tied input/output embeddings
- 4,096-token context, RoPE theta 500,000
- hybrid RoPE/NoPE attention, with NoPE attention every fourth layer
- digit-aware tokenizer
Usage
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4repo = "Asilarkness/Budgie-500m"
5tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
6model = AutoModelForCausalLM.from_pretrained(
7 repo,
8 trust_remote_code=True,
9 dtype=torch.bfloat16,
10).to("cuda").eval()
11
12messages = [{"role": "user", "content": "What is 17 * 24? Explain briefly."}]
13text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
14inputs = tokenizer(text, return_tensors="pt").to(model.device)
15output = model.generate(**inputs, max_new_tokens=160, do_sample=False)
16print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
trust_remote_code=True is required for the custom TestGeniy architecture and digit-aware tokenizer.
Prompting note
Reasoning is currently strongest with the native user/assistant template and no long system message. A descriptive identity system prompt improves dialogue identity but can reduce reasoning accuracy; do not compare system-prompted and bare runs as if they were the same protocol.
Limitations
Budgie remains an experimental 488M model and is still behind Qwen3-0.6B on the controlled composite. It can produce incorrect, truncated, repetitive, or poorly calibrated reasoning. Bare identity is weak. Small changes to weights or decoding can flip answers. The reported gates are controlled samples with uncertainty, not official leaderboard submissions.
Provenance
- Previous public revision:
39c877f07ddc80aa975809a4d87c848372316caf
- Alignment source:
Asilarkness/testgeniy
- Alignment source path:
candidates/budgie-alignment-v2/verified-math-a025
- Alignment source revision:
79aec5157c10e3f7e082823a6e9150482d740a91
- Released checkpoint:
verified-math-a025
- Model SHA-256:
7404de2c08df968f41800d1362704a0ad4106ea0116cd0cd4b1709612a2a66be
- Context length: 4,096 tokens
