You can deploy this model and start earning money today!

llaa33219/
MicroMixer-1-1M-TinyStories

Name: MicroMixer-1-1M-TinyStories
Availability: InStock

roneneldan/TinyStories

apache-2.0

Views

No views yet

MicroMixer-1 Logo

MicroMixer-1-1M-TinyStories

Parameters

Micro Language Model
Attention-Free • MLP-Only • Byte-Level

GitHub

📋 Overview

MicroMixer-1-1M is the largest model in the series with ~1M parameters. It can generate sentences like "Once upon a time there was a little girl named Lily" with reasonable fluency. Supports the longest sequence length of 256 tokens.

🏗️ Architecture

mermaid

graph TD
    A[Byte Input] --> B[Token Embedding]
    B --> C[RoPE Position Encoding]
    C --> D[ImprovedMixerLayer ×3]
    D --> E[LayerNorm]
    E --> F[LM Head]
    F --> G[Byte Output]
    
    style A fill:#007BFF,color:#fff
    style G fill:#00D620,color:#fff
    style D fill:#AE00FF,color:#fff

Model Configuration

Parameter	Value
Total Parameters	`967,584`
Hidden Dimension	`224`
Channel MLP Dimension	`576`
Number of Layers	`3`
Max Sequence Length	`256`
Vocabulary Size	`256` (Byte-level)

Core Components

┌─────────────────────────────────────────────┐
│           ImprovedMixerLayer                 │
│  ┌─────────────────────────────────────┐    │
│  │  LayerNorm → HyperMixing → Residual │    │ ← Token Mixing
│  ├─────────────────────────────────────┤    │
│  │  LayerNorm → MlpBlock → Residual    │    │ ← Channel Mixing
│  └─────────────────────────────────────┘    │
└─────────────────────────────────────────────┘

1️⃣ RoPE (Rotary Position Embedding)

Encodes positions via rotation transformations
Enables length extrapolation beyond training sequences

2️⃣ HyperMixing (Token Mixing)

Compresses past context via cumulative average pooling
Hypernetwork generates adaptive weights
O(S) complexity token mixing without attention

3️⃣ MlpBlock (Channel Mixing)

Non-linear transformation of feature dimensions
Structure: Linear → GELU → Linear

📈 Key Differences from 500K

Metric	500K	1M	Change
Parameters	557,328	967,584	1.7x
Hidden Dim	176	224	1.3x
Channel MLP	384	576	1.5x
Sequence Length	128	256	2x

🎯 Generation Examples

Prompt: "Once upon a time"
Output: "Once upon a time there was a little girl named Lily. She loved t..."

Prompt: "Hello"
Output: "Hellog ann he grit litle girls. She love loved in the"

📊 Model Comparison

Model	Parameters	Hidden	Seq Len	Quality
100K	136,908	84	64	⭐
300K	331,680	128	128	⭐⭐
500K	557,328	176	128	⭐⭐⭐
1M	967,584	224	256	⭐⭐⭐⭐

⚠️ Limitations

Limitation	Description
Grammatical Errors	Fully grammatical sentences still difficult
Name Instability	Same name varies: "Lily", "Limmy", "Amby"
Short Prompt Issues	"Hello", "The weather is" produce near-random output
Overfitting	Overfits to specific TinyStories phrases

📊 Training Data

Dataset: TinyStories

Simple children's stories dataset
Learns basic grammar and vocabulary
Contains many patterns like "Once upon a time", "little girl/boy"

🔧 Usage

python

import torch
from huggingface_hub import hf_hub_download
from src.model import MicroMixerV2, MicroMixerV2Config
from src.tokenizer import ByteTokenizer

# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-1.git
# cd MicroMixer-1

config = MicroMixerV2Config(
    max_seq_len=256,
    hidden_dim=224,
    channel_mlp_dim=576,
    num_layers=3,
    use_hyper=True,
)

model = MicroMixerV2(config)
weights_path = hf_hub_download("llaa33219/MicroMixer-1-1M-TinyStories", "model.pt")
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()

tokenizer = ByteTokenizer()
input_ids = torch.tensor([tokenizer.encode("Once upon a time")])

with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=64, temperature=0.8, top_k=40)

print(tokenizer.decode(output[0].tolist()))

GitHub

_{Part of the MicroMixer-1 research project}