MicroT-test1-50K-Discord-Dialogues is a 49,888-parameter vanilla decoder-only transformer — multi-head causal self-attention with RoPE — pretrained on Discord conversation data and fine-tuned with FMSP on 9,012 general-knowledge QA pairs.
This is the 50K member of the MicroT-test1 family: the registered attention-based reference baseline of the MicroMixer-4 project. MicroMixer-4's chassis is a pure MLP-Mixer with no attention; MicroT-test1 exists so that "Mixer vs Transformer at the same parameter count" is a fair, controlled comparison. It is deliberately boring — the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
At this size: below the memorization floor — full-988 EM is exactly 0, and fluency is partially degenerate.
🏗️ Architecture
mermaid
1graph TD
2 A[Byte Input]--> B[Embed 256→32]3 B --> C[Transformer Block × 3]4 C --> D[RMSNorm]5 D --> E[LM Head Tied with Embed]6 E --> F[Byte Output]78subgraph"Transformer Block (pre-norm)"9 X[Input 32]--> N1[RMSNorm]10 N1 --> AT["MHA 2 heads × d_head 16<br/>RoPE θ=10000 on q,k · causal SDPA"]11 AT --> R1[+ residual]12 R1 --> N2[RMSNorm]13 N2 --> MLP["GELU MLP 32→152→32"]14 MLP --> R2[+ residual]15end1617style A fill:#007BFF,color:#fff18style F fill:#00D620,color:#fff19style AT fill:#FF6600,color:#fff
Model Configuration
Parameter
Value
Total Parameters
49,888
Hidden Dimension (d_model)
32
Attention Heads
2 (d_head = 16 at every size)
Number of Blocks
3
FFN Hidden
152
Position Encoding
RoPE θ=10000 on q/k only (non-persistent buffers)
Attention
Causal MHA via F.scaled_dot_product_attention(is_causal=True)
Activation
GELU
Biases
None — no bias parameters anywhere
Normalization
RMSNorm (pre-norm)
Max Sequence Length
1024
Vocabulary Size
256 (byte-level)
Output Head
Tied with input embedding
Core Components
┌──────────────────────────────────────────────┐
│ Transformer Block (×3) │
│ h = h + MHA(RMSNorm(h)) # RoPE q/k, causal│
│ h = h + MLP(RMSNorm(h)) # GELU d→ffn→d │
│ no biases, no flash, no tricks — vanilla │
└──────────────────────────────────────────────┘
The d_head=16 contract is hard-asserted across all six sizes so that attention-head behavior is comparable at every budget and never confounds the memorization measurements.
🎯 Generation Examples
Questions the model was trained on (FMSP train set, 9,012 QA pairs — greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4):
[Prompt] User: Who painted the Mona Lisa?
Assistant:
[Output] The Sun is the capital city of the capital city of the largest of the computer of the comp…
fails — EM = 0; the trained fact is not retrievable at 50K
[Prompt] User: Who painted The Starry Night?
Assistant:
[Output] The Sun is the American in 1989 to 1915 and 1915 to 190 to 1901 and 1915 to 190 to 1901 an…
fails — numeric attractor loop instead of the trained answer
Questions the model has never seen and cannot answer (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):
[Prompt] User: Who composed the Symphony of Hollow Dawn?
Assistant:
[Output] The Sun is the Mark Africa is the most computer of the first computer in the 19th century …
fabricates — loop with real-word fragments
[Prompt] User: Who wrote the novel The Silent Marmalade?
Assistant:
[Output] The Sun is the American in 1998 and the computer of the American in 1999 and the American …
fabricates — same attractor as the trained questions
‡ where marked: degenerate-pass — the model does not engage the question at all, so there is nothing to hijack or fabricate with. Not boundary discipline.
49,888 parameters; capacity is the binding constraint on every axis
Reference baseline, not a product
Exists to score the Mixer against attention at matched budget
Knows only what it memorized
Knowledge is limited to the 9,012 trained QA pairs + Discord pretraining distribution
Does not abstain
Unknown questions are answered with fabrication or degeneration, not refusal — see the examples above
Byte-level noise
256-vocab byte tokenizer; PPL not comparable to BPE baselines
Research use only
Architecture/scaling research artifact, not a production model
🧬 Context
MicroT-test1 (V88) is the transformer control group of the MicroMixer-4 project — vanilla attention at the same six budgets as the V87 CCD-Mixer (MicroMixer-4 family), trained and evaluated under byte-identical recipes. The registered verdict: attention is the better raw LM and raw memorizer at ≥500K and holds boundary discipline one halving further down; the Mixer's defensible win is anti-collapse chatter fluency at the top end (family-record 0.912 at 500K). Full comparison: V88_README.md in the repository.
