MicroMixer-4-300K-Discord-Dialogues is a 292,525-parameter pure MLP-Mixer causal language model — no attention, no recurrence, no SSM — pretrained on Discord conversation data and then fine-tuned with FMSP (Fine-tuning with Minimal Parameter changes for Small-parameter LMs) on 9,012 general-knowledge QA pairs.
This is the 300K member of the MicroMixer-4 (V87 Final) family: the mid-size cut-off — chatter fluency survives, but memorization has fallen off the 500K→300K cliff and boundary discipline is gone.
The backbone is V87 Final, the project's champion architecture — a CCD-Mixer (Content-gated mixture of shared-weight Dilated convolutions) crowned overall champion of the 1M architecture census (V86), frozen as the final chassis and scaled to six parameter budgets. The 300K preset reproduces the champion recipe verbatim at its budget.
🏗️ Architecture
mermaid
1graph TD
2 A[Byte Input]--> B[Embed 256→80 NoPE]3 B --> C[CCD-Mixer Block × 5]4 C --> D[RMSNorm]5 D --> E[LM Head Tied with Embed]6 E --> F[Byte Output]78subgraph"CCD-Mixer Block"9 X[Input 80]--> U["Linear d→2d → split v, g"]10 U --> RP[Full RoPE on v AND g]11 RP --> M["Shared-weight dilated conv<br/>dilations 1·2·4·8, k=81"]12 M --> G["Per-position 4-way gate<br/>softmax(Linear_dil(x)/τ)"]13 G --> O["W_o(v ⊙ g) — zero-init"]14 O --> SW[SwiGLU Channel-Mix]15 SW --> RM[ReMixerLayer sidecar]16end1718style A fill:#007BFF,color:#fff19style F fill:#00D620,color:#fff20style G fill:#AE00FF,color:#fff21style M fill:#FF6600,color:#fff
Model Configuration
Parameter
Value
Total Parameters
292,525
Hidden Dimension (d_model)
80
Number of Blocks
5
Token-Mix
GLCTokenMixCCD (content-gated mixture of shared-weight dilated causal conv)
Dilations
(1, 2, 4, 8) — one shared depthwise kernel, zero extra conv params
Depthwise Kernel Size
81
RoPE
Full RoPE on both v and g (V76 "RPG" pattern)
Channel-Mix
SwiGLU
Sidecar
ReMixerLayer per block (label_dim 16, pool_heads 4)
Max Sequence Length
1024
Vocabulary Size
256 (byte-level)
Position Encoding
RoPE inside token-mix only; no position embedding table
Normalization
RMSNorm (pre-norm)
Output Head
Tied with input embedding
Zero-Init
W_o, dil_gate, log_τ — silent at init
Core Components
┌──────────────────────────────────────────────────────────────┐
│ CCD-Mixer Block (×5) │
│ u = Linear(d → 2d)(x) │
│ v, g = u.chunk(2) │
│ v = RoPE(v) g = RoPE(g) ← full-RoPE (RPG) │
│ y_d = CausalDSConv(v, dilation=d) for d ∈ (1,2,4,8) │
│ └── ONE shared depthwise kernel │
│ w(t) = softmax(Linear_dil(x)_t / τ) ← per-position │
│ v = Σ_d w_d(t) · y_d(t) time-varying filter │
│ out = W_o(v ⊙ g) ← W_o zero-init │
│ then SwiGLU channel-mix + ReMixerLayer sidecar │
└──────────────────────────────────────────────────────────────┘
The token-mix is non-LTI (time-varying): the per-position gate remixes four dilated views of the same kernel at every byte, which is the mechanism that breaks the periodic-orbit collapse that pure LTI mixers fall into — without attention and without a position table.
🎯 Generation Examples
Questions the model was trained on (FMSP train set, 9,012 QA pairs — greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4):
[Prompt] User: Who painted Girl with a Pearl Earring?
Assistant:
[Output] Johannes Vermeer painted Girl with a Pearl Earring around 1665, and it is often called the…
correct — one of the ~56 facts this size retains at generation-verify strength
[Prompt] User: Who painted the Mona Lisa?
Assistant:
[Output] The Sumerianship Chismerical label painted Mascule around 1815, and it is the taller syste…
wrong — past the 500K→300K memorization cliff, most trained facts come back garbled
Questions the model has never seen and cannot answer (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):
[Prompt] User: Who painted the Glimmering Frostberry?
Assistant:
[Output] Foreign Marcoding Home Pentrack Waterfause, or the early 197l-burst meaning mathematician…
fabricates — word salad on out-of-distribution input
[Prompt] User: Who wrote the novel The Silent Marmalade?
Assistant:
[Output] Martin Lunge wrote the novel 'Les Misenlerica.
fabricates — invents a plausible-sounding author and title
‡ where marked: degenerate-pass — the model does not engage the question at all, so there is nothing to hijack or fabricate with. Not boundary discipline.
292,525 parameters; capacity is the binding constraint on every axis
Knows only what it memorized
Knowledge is limited to the 9,012 trained QA pairs + Discord pretraining distribution
Does not abstain
Unknown questions are answered with fabrication or degeneration, not refusal — see the examples above
Byte-level noise
256-vocab byte tokenizer; PPL not comparable to BPE baselines
Research use only
Architecture/scaling research artifact, not a production model
🧬 Context
MicroMixer-4 is the fourth generation of the MicroMixer research line: sub-1M-parameter language models built purely from MLP-Mixer operations. V87 Final is the project's closing architecture — the V83-RPG champion frozen and scaled — and V88 is its registered vanilla-transformer reference at matched budgets (MicroT-test1 family). Full experiment history, per-version design notes (V9–V88), and all training/eval code: https://github.com/llaa33219/MicroMixer-4.
