MicroT-test1-10K-TinyStories is a 9,808-parameter vanilla decoder-only transformer — multi-head causal self-attention with RoPE — pretrained on TinyStories as part of the MicroMixer-4 dataset-efficiency comparison study (analysis). TinyStories is a corpus of ~200K short synthetic children's stories (GPT-3.5/4-generated, constrained grammar and vocabulary).
⚠️ This repo is pretrain-only — it is NOT FMSP-fine-tuned. TinyStories contains free-flowing prose with no User:/Assistant: dialogue markers, so the FMSP answer-only cross-entropy recipe does not apply (there is no answer region to isolate). What you get is the raw pretrained backbone (seed 42, V76 recipe, 3 epochs). It continues stories; it does not answer questions or follow instructions.
It is the registered attention-based reference baseline — a deliberately boring standard 2018–2020 transformer recipe (no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE). The 10K preset reproduces the champion recipe at its budget.
🏗️ Architecture
mermaid
1graph TD
2 A[Byte Input]--> B[Embed 256→16]3 B --> C[Transformer Block × 2]4 C --> D[RMSNorm]5 D --> E[LM Head Tied with Embed]6 E --> F[Byte Output]78subgraph"Transformer Block (pre-norm)"9 X[Input 16]--> N1[RMSNorm]10 N1 --> AT["MHA 1 heads × d_head 16<br/>RoPE θ=10000 on q,k · causal SDPA"]11 AT --> R1[+ residual]12 R1 --> N2[RMSNorm]13 N2 --> MLP["GELU MLP 16→56→16"]14 MLP --> R2[+ residual]15end1617style A fill:#007BFF,color:#fff18style F fill:#00D620,color:#fff19style AT fill:#FF6600,color:#fff
Model Configuration
Parameter
Value
Hidden Dimension (d_model)
16
Attention Heads
1 (d_head = 16 at every size)
Number of Blocks
2
FFN Hidden
56
Position Encoding
RoPE θ=10000 on q/k only (non-persistent buffers)
Attention
Causal MHA via F.scaled_dot_product_attention(is_causal=True)
Activation
GELU
Biases
None — no bias parameters anywhere
Normalization
RMSNorm (pre-norm)
Max Sequence Length
1024
Vocabulary Size
256 (byte-level)
Output Head
Tied with input embedding
Core Components
┌──────────────────────────────────────────────┐
│ Transformer Block (×2) │
│ h = h + MHA(RMSNorm(h)) # RoPE q/k, causal│
│ h = h + MLP(RMSNorm(h)) # GELU d→ffn→d │
│ no biases, no flash, no tricks — vanilla │
└──────────────────────────────────────────────┘
The d_head=16 contract is hard-asserted across all six sizes so that attention-head behavior is comparable at every budget and never confounds the memorization measurements.
🎯 Generation Examples
Story-continuation probes (raw text — no User:/Assistant: prefix, since TinyStories is a plain LM; greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4, max_new_tokens=200):
[Prompt] Once upon a time, there was a little girl named Lily.
[Output] She was to the said, "I was so the said, "I was so the said, "I was so the said, "I was so the …
degenerate — collapses into word-salad repetition at this size
[Prompt] One day, a boy named Tom found a big red balloon.
[Output] They was so the said, "I was so the said, "I was so the said, "I was so the said, "I was so the…
degenerate — collapses into word-salad repetition at this size
[Prompt] The sun was shining and the birds were singing.
[Output] She was so the said to she was so the said, "I was so the said, "I was so the said, "I was so t…
degenerate — collapses into word-salad repetition at this size
[Prompt] A little dog ran into the garden and saw
[Output] a ain her was a saw the said, "I was so the said, "I was so the said, "I was so the said, "I wa…
degenerate — collapses into word-salad repetition at this size
📊 Results
Pretraining (TinyStories, V76 recipe, 3 epochs)
Metric
1 ep
2 ep
3 ep
Val PPL
4.72
4.51
4.31
AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42 · plain CE on non-pad bytes.
MicroT-test1 TinyStories family (pretrain-only, all sizes)
Size
Params
3ep Val PPL
1M
996,736
1.69
500K
498,528
1.79
300K
297,680
1.90
100K
97,872
2.28
50K
49,888
2.60
10K
9,808
4.31
Pretrain-only family — FMSP-based axes (chatter fluency, full-988 EM, q-relevance, OOD, unanswerable fabrication) are N/A: TinyStories has no User:/Assistant: markers, so the answer-only-CE recipe does not apply.
📚 Training Data
Pretraining: TinyStories — synthetic short stories generated by GPT-3.5/4 with a constrained vocabulary and simple grammar, ~200K stories sampled, flattened to 1024-byte sequences, 3 epochs. No User:/Assistant: dialogue structure.
🔧 Usage
Files in this repository
epoch_{0,1,2}.safetensors — per-epoch pretrained backbone weights (pickle-free safetensors). epoch_2.safetensors is the final (3rd-epoch) checkpoint. No FMSP adapter — this is the plain backbone.
Load and generate (local clone)
python
1import torch
2from safetensors.torch import load_file
3from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_10k
4from src.tokenizer import ByteTokenizer
56# Clone the code repository first:7# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-489cfg = v88_transformer_10k()10model = MicroMixerV88Transformer(cfg)# plain backbone — NO attach_adapter (pretrain-only)11model.load_state_dict(load_file("epoch_2.safetensors"), strict=True)12model.eval()1314tok = ByteTokenizer()15prompt ="Once upon a time, there was a little girl named Lily."16ids = tok.encode(prompt)17if ids and ids[-1]== tok.eos_token_id:18 ids = ids[:-1]# ByteTokenizer appends EOS; the prompt must end open19ids = torch.tensor([ids])20with torch.no_grad():21 out = model.generate(22 ids, max_new_tokens=200,23 temperature=0.0,# greedy24 repetition_penalty=1.2,25 no_repeat_ngram_size=4,26 eos_token_id=tok.eos_token_id,27)28print(prompt + tok.decode(out[0].tolist()[len(ids):]))
Note the differences from the FMSP cards: (1) no attach_adapter — the backbone is loaded
as-is; (2) the prompt is raw story text, not the User: …\n\nAssistant: dialogue format.
Load from Hugging Face Hub (no clone of the weights needed)
python
1import torch
2from huggingface_hub import hf_hub_download
3from safetensors.torch import load_file
4from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_10k
56REPO ="llaa33219/MicroT-test1-10K-TinyStories"78cfg = v88_transformer_10k()9model = MicroMixerV88Transformer(cfg)10model.load_state_dict(11 load_file(hf_hub_download(REPO,"epoch_2.safetensors")), strict=True)12model.eval()13# ... continue a story as above
⚠️ Limitations
Limitation
Description
Pretrain-only — no instruction/QA ability
Not FMSP-fine-tuned; it only continues TinyStories-style prose. It cannot answer questions or follow instructions.
Micro parameters
9,808 parameters; capacity is the binding constraint
Knows only TinyStories
Distribution is synthetic children's stories; no real-world knowledge
Byte-level noise
256-vocab byte tokenizer; PPL not comparable to BPE baselines
Research use only
Architecture/pretraining research artifact, not a production model
🧬 Context
This is the 10K TinyStories-pretrained arm of the dataset-efficiency comparison study in the MicroMixer-4 project — the pretrain-only third corpus alongside the UltraChat and SmolTalk2 FMSP arms (TinyStories is excluded from the FMSP/eval battery because it has no User:/Assistant: markers). Sibling repos: llaa33219/{MicroMixer-4,MicroT-test1}-{1M..10K}-TinyStories, plus the UltraChat/SmolTalk2 arms …-{UltraChat,SmolTalk2} and the discord-pretrained baselines llaa33219/{MicroMixer-4,MicroT-test1}-{1M..10K}. Full analysis: DATASET_COMPARISON_ANALYSIS.md.
