MicroMixer-4-500K-TinyStories is a 491,742-parameter pure MLP-Mixer causal language model — no attention, no recurrence, no SSM — pretrained on TinyStories as part of the MicroMixer-4 dataset-efficiency comparison study (analysis). TinyStories is a corpus of ~200K short synthetic children's stories (GPT-3.5/4-generated, constrained grammar and vocabulary).
⚠️ This repo is pretrain-only — it is NOT FMSP-fine-tuned. TinyStories contains free-flowing prose with no User:/Assistant: dialogue markers, so the FMSP answer-only cross-entropy recipe does not apply (there is no answer region to isolate). What you get is the raw pretrained backbone (seed 42, V76 recipe, 3 epochs). It continues stories; it does not answer questions or follow instructions.
The backbone is V87 Final, the project's champion CCD-Mixer architecture (the V83-RPG champion frozen and scaled to six budgets). The 500K preset reproduces the champion recipe at its budget.
🏗️ Architecture
mermaid
1graph TD
2 A[Byte Input]--> B[Embed 256→96 NoPE]3 B --> C[CCD-Mixer Block × 6]4 C --> D[RMSNorm]5 D --> E[LM Head Tied with Embed]6 E --> F[Byte Output]78subgraph"CCD-Mixer Block"9 X[Input 96]--> U["Linear d→2d → split v, g"]10 U --> RP[Full RoPE on v AND g]11 RP --> M["Shared-weight dilated conv<br/>dilations 1·2·4·8, k=97"]12 M --> G["Per-position 4-way gate<br/>softmax(Linear_dil(x)/τ)"]13 G --> O["W_o(v ⊙ g) — zero-init"]14 O --> SW[SwiGLU Channel-Mix]15 SW --> RM[ReMixerLayer sidecar]16end1718style A fill:#007BFF,color:#fff19style F fill:#00D620,color:#fff20style G fill:#AE00FF,color:#fff21style M fill:#FF6600,color:#fff
Model Configuration
Parameter
Value
Hidden Dimension (d_model)
96
Number of Blocks
6
Token-Mix
GLCTokenMixCCD (content-gated mixture of shared-weight dilated causal conv)
Dilations
(1, 2, 4, 8) — one shared depthwise kernel, zero extra conv params
Depthwise Kernel Size
97
RoPE
Full RoPE on both v and g (V76 "RPG" pattern)
Channel-Mix
SwiGLU
Sidecar
ReMixerLayer per block (label_dim 16, pool_heads 4)
Max Sequence Length
1024
Vocabulary Size
256 (byte-level)
Position Encoding
RoPE inside token-mix only; no position embedding table
Normalization
RMSNorm (pre-norm)
Output Head
Tied with input embedding
Zero-Init
W_o, dil_gate, log_τ — silent at init
Core Components
┌──────────────────────────────────────────────────────────────┐
│ CCD-Mixer Block (×6) │
│ u = Linear(d → 2d)(x) │
│ v, g = u.chunk(2) │
│ v = RoPE(v) g = RoPE(g) ← full-RoPE (RPG) │
│ y_d = CausalDSConv(v, dilation=d) for d ∈ (1,2,4,8) │
│ └── ONE shared depthwise kernel │
│ w(t) = softmax(Linear_dil(x)_t / τ) ← per-position │
│ v = Σ_d w_d(t) · y_d(t) time-varying filter │
│ out = W_o(v ⊙ g) ← W_o zero-init │
│ then SwiGLU channel-mix + ReMixerLayer sidecar │
└──────────────────────────────────────────────────────────────┘
The token-mix is non-LTI (time-varying): the per-position gate remixes four dilated views of the same kernel at every byte, which is the mechanism that breaks the periodic-orbit collapse that pure LTI mixers fall into — without attention and without a position table.
🎯 Generation Examples
Story-continuation probes (raw text — no User:/Assistant: prefix, since TinyStories is a plain LM; greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4, max_new_tokens=200):
[Prompt] Once upon a time, there was a little girl named Lily.
[Output] She loved to play outside in the sunshine. One day, she saw a big tree with a big stick on it. …
coherent — on-topic story continuation in the TinyStories register
[Prompt] One day, a boy named Tom found a big red balloon.
[Output] He wanted to play with it, but he was very sad. He did not know what to do. He wanted to play w…
coherent — on-topic story continuation in the TinyStories register
[Prompt] The sun was shining and the birds were singing.
[Output] The bird was so excited to see the birds and the birds singing and the birds were singing. They…
drifts — starts as story but falls into a repetitive loop
[Prompt] A little dog ran into the garden and saw
[Output] a big box of leaves. He wanted to get it out and see what was inside. He was so excited to see …
coherent — on-topic story continuation in the TinyStories register
📊 Results
Pretraining (TinyStories, V76 recipe, 3 epochs)
Metric
1 ep
2 ep
3 ep
Val PPL
2.07
2.01
1.90
AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42 · plain CE on non-pad bytes.
MicroMixer-4 TinyStories family (pretrain-only, all sizes)
Size
Params
3ep Val PPL
1M
996,873
1.78
500K
491,742
1.90
300K
292,525
2.00
100K
95,084
2.31
50K
48,684
2.60
10K
9,666
4.06
Pretrain-only family — FMSP-based axes (chatter fluency, full-988 EM, q-relevance, OOD, unanswerable fabrication) are N/A: TinyStories has no User:/Assistant: markers, so the answer-only-CE recipe does not apply.
📚 Training Data
Pretraining: TinyStories — synthetic short stories generated by GPT-3.5/4 with a constrained vocabulary and simple grammar, ~200K stories sampled, flattened to 1024-byte sequences, 3 epochs. No User:/Assistant: dialogue structure.
🔧 Usage
Files in this repository
epoch_{0,1,2}.safetensors — per-epoch pretrained backbone weights (pickle-free safetensors). epoch_2.safetensors is the final (3rd-epoch) checkpoint. No FMSP adapter — this is the plain backbone.
Load and generate (local clone)
python
1import torch
2from safetensors.torch import load_file
3from src.model_v87_final import MicroMixerV87Final, v87_final_500k
4from src.tokenizer import ByteTokenizer
56# Clone the code repository first:7# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-489cfg = v87_final_500k()10model = MicroMixerV87Final(cfg)# plain backbone — NO attach_adapter (pretrain-only)11model.load_state_dict(load_file("epoch_2.safetensors"), strict=True)12model.eval()1314tok = ByteTokenizer()15prompt ="Once upon a time, there was a little girl named Lily."16ids = tok.encode(prompt)17if ids and ids[-1]== tok.eos_token_id:18 ids = ids[:-1]# ByteTokenizer appends EOS; the prompt must end open19ids = torch.tensor([ids])20with torch.no_grad():21 out = model.generate(22 ids, max_new_tokens=200,23 temperature=0.0,# greedy24 repetition_penalty=1.2,25 no_repeat_ngram_size=4,26 eos_token_id=tok.eos_token_id,27)28print(prompt + tok.decode(out[0].tolist()[len(ids):]))
Note the differences from the FMSP cards: (1) no attach_adapter — the backbone is loaded
as-is; (2) the prompt is raw story text, not the User: …\n\nAssistant: dialogue format.
Load from Hugging Face Hub (no clone of the weights needed)
python
1import torch
2from huggingface_hub import hf_hub_download
3from safetensors.torch import load_file
4from src.model_v87_final import MicroMixerV87Final, v87_final_500k
56REPO ="llaa33219/MicroMixer-4-500K-TinyStories"78cfg = v87_final_500k()9model = MicroMixerV87Final(cfg)10model.load_state_dict(11 load_file(hf_hub_download(REPO,"epoch_2.safetensors")), strict=True)12model.eval()13# ... continue a story as above
⚠️ Limitations
Limitation
Description
Pretrain-only — no instruction/QA ability
Not FMSP-fine-tuned; it only continues TinyStories-style prose. It cannot answer questions or follow instructions.
Micro parameters
491,742 parameters; capacity is the binding constraint
Knows only TinyStories
Distribution is synthetic children's stories; no real-world knowledge
Byte-level noise
256-vocab byte tokenizer; PPL not comparable to BPE baselines
Research use only
Architecture/pretraining research artifact, not a production model
🧬 Context
This is the 500K TinyStories-pretrained arm of the dataset-efficiency comparison study in the MicroMixer-4 project — the pretrain-only third corpus alongside the UltraChat and SmolTalk2 FMSP arms (TinyStories is excluded from the FMSP/eval battery because it has no User:/Assistant: markers). Sibling repos: llaa33219/{MicroMixer-4,MicroT-test1}-{1M..10K}-TinyStories, plus the UltraChat/SmolTalk2 arms …-{UltraChat,SmolTalk2} and the discord-pretrained baselines llaa33219/{MicroMixer-4,MicroT-test1}-{1M..10K}. Full analysis: DATASET_COMPARISON_ANALYSIS.md.
