MicroT-test1-300K-SmolTalk2 is a 297,680-parameter vanilla decoder-only transformer — multi-head causal self-attention with RoPE — pretrained on SmolTalk2 conversation data (instead of the project's Discord-Dialogues baseline) and then fine-tuned with FMSP on 9,012 general-knowledge QA pairs.
This is the 300K member of the MicroT-test1 family: the registered attention-based reference baseline of the MicroMixer-4 project, here rerun on SmolTalk2 as part of the dataset-efficiency comparison study — six parameter budgets × two architectures × two open pretraining corpora, all fine-tuned with the identical P05 FMSP recipe at seed 42. Analysis.
It is deliberately boring — the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
🏗️ Architecture
mermaid
1graph TD
2 A[Byte Input]--> B[Embed 256→80]3 B --> C[Transformer Block × 4]4 C --> D[RMSNorm]5 D --> E[LM Head Tied with Embed]6 E --> F[Byte Output]78subgraph"Transformer Block (pre-norm)"9 X[Input 80]--> N1[RMSNorm]10 N1 --> AT["MHA 5 heads × d_head 16<br/>RoPE θ=10000 on q,k · causal SDPA"]11 AT --> R1[+ residual]12 R1 --> N2[RMSNorm]13 N2 --> MLP["GELU MLP 80→272→80"]14 MLP --> R2[+ residual]15end1617style A fill:#007BFF,color:#fff18style F fill:#00D620,color:#fff19style AT fill:#FF6600,color:#fff
Model Configuration
Parameter
Value
Hidden Dimension (d_model)
80
Attention Heads
5 (d_head = 16 at every size)
Number of Blocks
4
FFN Hidden
272
Position Encoding
RoPE θ=10000 on q/k only (non-persistent buffers)
Attention
Causal MHA via F.scaled_dot_product_attention(is_causal=True)
Activation
GELU
Biases
None — no bias parameters anywhere
Normalization
RMSNorm (pre-norm)
Max Sequence Length
1024
Vocabulary Size
256 (byte-level)
Output Head
Tied with input embedding
Core Components
┌──────────────────────────────────────────────┐
│ Transformer Block (×4) │
│ h = h + MHA(RMSNorm(h)) # RoPE q/k, causal│
│ h = h + MLP(RMSNorm(h)) # GELU d→ffn→d │
│ no biases, no flash, no tricks — vanilla │
└──────────────────────────────────────────────┘
The d_head=16 contract is hard-asserted across all six sizes so that attention-head behavior is comparable at every budget and never confounds the memorization measurements.
🎯 Generation Examples
Questions the model was trained on (FMSP train set, 9,012 QA pairs — greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4):
[Prompt] User: Who painted the Mona Lisa?
Assistant:
[Output] Leonardo da Vinci painted the Mona Lisa, an iconic portrait completed in the early 16th century…
correct — near-verbatim recitation of the trained fact
[Prompt] User: Who painted The Starry Night?
Assistant:
[Output] George Watt composed the start at New World Wide at a thypoe in 1876.
fabricates — gives a wrong answer to a trained question at this size
Questions the model has never seen and cannot answer (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):
[Prompt] User: Who painted the Glimmering Frostberry?
Assistant:
[Output] Glob Florence is the largest in 1966 on the result between Floror Graceric pi and nutrients in …
fabricates — plausible-sounding nonsense on a nonexistent subject
[Prompt] User: Who composed the Symphony of Hollow Dawn?
Assistant:
[Output] The Scottish backstory is a marketing phanchise, part of which 180 kilometers remainders backi…
fabricates — plausible-sounding nonsense on a nonexistent subject
📊 Results
Pretraining (SmolTalk2, V76 recipe, 3 epochs)
Metric
1 ep
2 ep
3 ep
Val PPL
2.85
2.73
2.53
Identical recipe to the MicroMixer-4 mixer: AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42.
Single-seed run (seed 42); the discord-pretrained cards report a 3-seed mean for Full-988 EM. ‡ where marked: degenerate-pass.
MicroT-test1 SmolTalk2 family (same protocol, all sizes)
Size
Params
3ep Val PPL
Chatter d2
Full-988 EM
qrel echo/hijack
OOD hijack
1M
996,736
2.12
0.911
744
100.0 / 0.0
37.3%
500K
498,528
2.32
0.932
580
86.0 / 7.0
42.4%
300K
297,680
2.53
0.775
137
55.0 / 26.0
40.7%
100K
97,872
3.13
0.673
0
5.0 / 86.0
55.9%
50K
49,888
3.68
0.549
0
8.0 / 49.0
50.8%
10K
9,808
6.04
0.447
0
0.0 / 0.0
0.0%
Seed-42 single runs (pretrained on SmolTalk2; the discord-pretrained families report 3-seed EM means).
📚 Training Data
Pretraining: SmolTalk2 — 200K-cap sample of the smoltalk_smollm3_smol_magpie_ultra_no_think split (200K of 406,843; Apache 2.0), ~6-turn multi-turn conversations with no reasoning traces and system messages excluded, flattened to User:/Assistant: format, 1024-byte sequences, 3 epochs.
297,680 parameters; capacity is the binding constraint on every axis
Reference baseline, not a product
Exists to score the Mixer against attention at matched budget
Knows only what it memorized
Knowledge is limited to the 9,012 trained QA pairs + the pretraining-corpus distribution
Does not abstain
Unknown questions are answered with fabrication or degeneration, not refusal — see the examples above
Byte-level noise
256-vocab byte tokenizer; PPL not comparable to BPE baselines
Research use only
Architecture/scaling research artifact, not a production model
🧬 Context
This is the 300K SmolTalk2-pretrained arm of the dataset-efficiency comparison study (24 arms: {V87, V88} × {UltraChat, SmolTalk2} × six sizes) in the MicroMixer-4 project. Each arm swaps the Discord-Dialogues pretraining baseline for an open corpus (SmolTalk2) and reruns the identical P05 FMSP recipe at seed 42. Sibling repos: llaa33219/MicroMixer-4-{1M..10K}-{UltraChat,SmolTalk2} and llaa33219/MicroT-test1-{1M..10K}-{UltraChat,SmolTalk2}; the discord-pretrained baselines are llaa33219/MicroMixer-4-{1M..10K} and llaa33219/MicroT-test1-{1M..10K}. Full analysis: DATASET_COMPARISON_ANALYSIS.md.
