MicroT-test1-1M-Discord-Dialogues is a 996,736-parameter vanilla decoder-only transformer — multi-head causal self-attention with RoPE — pretrained on Discord conversation data and fine-tuned with FMSP on 9,012 general-knowledge QA pairs.
This is the 1M member of the MicroT-test1 family: the registered attention-based reference baseline of the MicroMixer-4 project. MicroMixer-4's chassis is a pure MLP-Mixer with no attention; MicroT-test1 exists so that "Mixer vs Transformer at the same parameter count" is a fair, controlled comparison. It is deliberately boring — the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.
At this size: the flagship — the only size that holds all four capability axes at once (fluency, memorization, boundary discipline, language preservation).
🏗️ Architecture
mermaid
1graph TD
2 A[Byte Input]--> B[Embed 256→128]3 B --> C[Transformer Block × 5]4 C --> D[RMSNorm]5 D --> E[LM Head Tied with Embed]6 E --> F[Byte Output]78subgraph"Transformer Block (pre-norm)"9 X[Input 128]--> N1[RMSNorm]10 N1 --> AT["MHA 8 heads × d_head 16<br/>RoPE θ=10000 on q,k · causal SDPA"]11 AT --> R1[+ residual]12 R1 --> N2[RMSNorm]13 N2 --> MLP["GELU MLP 128→496→128"]14 MLP --> R2[+ residual]15end1617style A fill:#007BFF,color:#fff18style F fill:#00D620,color:#fff19style AT fill:#FF6600,color:#fff
Model Configuration
Parameter
Value
Total Parameters
996,736
Hidden Dimension (d_model)
128
Attention Heads
8 (d_head = 16 at every size)
Number of Blocks
5
FFN Hidden
496
Position Encoding
RoPE θ=10000 on q/k only (non-persistent buffers)
Attention
Causal MHA via F.scaled_dot_product_attention(is_causal=True)
Activation
GELU
Biases
None — no bias parameters anywhere
Normalization
RMSNorm (pre-norm)
Max Sequence Length
1024
Vocabulary Size
256 (byte-level)
Output Head
Tied with input embedding
Core Components
┌──────────────────────────────────────────────┐
│ Transformer Block (×5) │
│ h = h + MHA(RMSNorm(h)) # RoPE q/k, causal│
│ h = h + MLP(RMSNorm(h)) # GELU d→ffn→d │
│ no biases, no flash, no tricks — vanilla │
└──────────────────────────────────────────────┘
The d_head=16 contract is hard-asserted across all six sizes so that attention-head behavior is comparable at every budget and never confounds the memorization measurements.
🎯 Generation Examples
Questions the model was trained on (FMSP train set, 9,012 QA pairs — greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4):
[Prompt] User: Who painted the Mona Lisa?
Assistant:
[Output] Leonardo da Vinci painted the Mona Lisa during the early 1500s in Italy.
correct — near-verbatim recitation of the trained fact
[Prompt] User: Who painted The Last Supper?
Assistant:
[Output] Leonardo da Vinci painted The Last Supper between 1495 and 1498 on the wall of the refecto…
correct — recites the trained answer to the truncation limit
Questions the model has never seen and cannot answer (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):
[Prompt] User: Who painted the Glimmering Frostberry?
Assistant:
[Output] Glimbre5 stands for Glimsrom Glimbreydropic, are pretending to be Standard President over …
fabricates — invents a definition for a nonexistent painting
[Prompt] User: Who composed the Symphony of Hollow Dawn?
Assistant:
[Output] Dani Australia (Teky Koi) is the completion of Koi aap koi raha raha bhi bhi nahi baat ko …
fabricates — slips into multilingual noise on out-of-distribution input
‡ where marked: degenerate-pass — the model does not engage the question at all, so there is nothing to hijack or fabricate with. Not boundary discipline.
996,736 parameters; capacity is the binding constraint on every axis
Reference baseline, not a product
Exists to score the Mixer against attention at matched budget
Knows only what it memorized
Knowledge is limited to the 9,012 trained QA pairs + Discord pretraining distribution
Does not abstain
Unknown questions are answered with fabrication or degeneration, not refusal — see the examples above
Byte-level noise
256-vocab byte tokenizer; PPL not comparable to BPE baselines
Research use only
Architecture/scaling research artifact, not a production model
🧬 Context
MicroT-test1 (V88) is the transformer control group of the MicroMixer-4 project — vanilla attention at the same six budgets as the V87 CCD-Mixer (MicroMixer-4 family), trained and evaluated under byte-identical recipes. The registered verdict: attention is the better raw LM and raw memorizer at ≥500K and holds boundary discipline one halving further down; the Mixer's defensible win is anti-collapse chatter fluency at the top end (family-record 0.912 at 500K). Full comparison: V88_README.md in the repository.
