You can deploy this model and start earning money today!

llaa33219/
MicroMixer-3-300K-discord-dialogues

Name: MicroMixer-3-300K-discord-dialogues
Availability: InStock

factorized-state-content

mookiezi/Discord-Dialogues

apache-2.0

Views

No views yet

MicroMixer-3 Logo

MicroMixer-3-300K-discord-dialogues

Parameters

Micro Language Model
Attention-Free • MLP-Only • Byte-Level • Factorized State-Content

GitHub

📋 Overview

MicroMixer-3-300K-discord-dialogues is a ~277K parameter Factorized State-Content MLP-Mixer (FSC-Mixer) language model trained on Discord conversation data. The 300K variant uses a 6-layer block structure (vs 8 in the 500K / 1M variants) and a shorter state-dilation schedule (1,2,4,8,16,32) — a capacity-friendly trade-off that still keeps the full state-vs-content factorization.

🏗️ Architecture

mermaid

graph TD
    A[Byte Input] --> B[Embed 256→80 NoPE]
    B --> C[FSC-Mixer Block × 6]
    C --> D[RMSNorm]
    D --> E[LM Head Tied with Embed]
    E --> F[Byte Output]

    subgraph "FSC-Mixer Block"
        X[Input 80] --> Split
        Split --> Cc[Content 40]
        Split --> Cs[State 40]

        Cc --> RN1[RMSNorm] --> CTM[CausalDSConv1d k=3 dil=1]
        CTM --> CCM[Channel MLP 4×]
        CCM --> Cc2[Content Out]

        Cs --> RN2[RMSNorm] --> STM[CausalDSConv1d k=3 dil=d_l]
        STM --> SCM[Channel MLP 2×]
        SCM --> Cs2[State Out]

        Cc2 --> GateRecomb
        Cs2 --> GateRecomb
        GateRecomb["g⊙c + (1-g)⊙W_s@s"] --> Out[80 concat]
    end

    style A fill:#007BFF,color:#fff
    style F fill:#00D620,color:#fff
    style GateRecomb fill:#AE00FF,color:#fff
    style CTM fill:#FF6600,color:#fff
    style STM fill:#FF6600,color:#fff

Model Configuration

Parameter	Value
Total Parameters	`277,120`
Hidden Dimension (d_model)	`80`
Content Dimension (d_content)	`40`
State Dimension (d_state)	`40`
Number of Layers	`6`
State Dilation Schedule	`(1, 2, 4, 8, 16, 32)`
Content Dilation	`1` (local)
State Receptive Field	`127 bytes` by layer 6
Content Channel MLP Expansion	`4×`
State Channel MLP Expansion	`2×`
Max Sequence Length	`1024`
Vocabulary Size	`256` (Byte-level)
Position Encoding	`NoPE` (causal structure provides implicit position)
Activation	`GELU`
Normalization	`RMSNorm`

Core Components

┌────────────────────────────────────────────────────┐
│              FSC-Mixer Block (×6)                   │
│  ┌──────────────────────────────────────────┐      │
│  │  Content Branch                          │      │
│  │  RMSNorm → CausalDSConv1d(k=3,d=1) → +  │      │ ← Local morphology
│  │  Channel MLP (4×) → +                    │      │
│  ├──────────────────────────────────────────┤      │
│  │  State Branch                            │      │
│  │  RMSNorm → CausalDSConv1d(k=3,d=d_l) → + │      │ ← Long-range syntax
│  │  Channel MLP (2×) → +                    │      │   (dilations exponentially)
│  ├──────────────────────────────────────────┤      │
│  │  State-Gated Recombination               │      │
│  │  g = σ(Linear_s(s))                      │      │ ← Attention equivalent
│  │  out = g⊙c + (1-g)⊙(W_s@s)               │      │   (linear + sigmoid)
│  └──────────────────────────────────────────┘      │
└────────────────────────────────────────────────────┘

1️⃣ Causal Depthwise-Separable Conv (Token Mixing)

Content branch: dilation=1, captures local morphology (3-byte window)
State branch: dilations grow exponentially (1,2,4,8,16,32), reaching a 127-byte receptive field by layer 6
Pure convolution → fully parallel across the time dim, no Python loops

2️⃣ Channel MLPs

Content: Linear → GELU → Linear with 4× expansion
State: Linear → GELU → Linear with 2× expansion (smaller, because state is meant to be a "summary")

3️⃣ State-Gated Recombination (MLP-Mixer "Attention Equivalent")

g = σ(Linear_s(s)) — gate computed from the state branch
out = g ⊙ c + (1-g) ⊙ (W_s @ s) — state modulates content via a learned, content-dependent gate
No Q·K^T scores, no O(n) state update — all linear + sigmoid (true MLP)

🎯 Generation Examples

Note: All four V8 FSC checkpoints (100K / 300K / 500K / 1M) were trained for 3 epochs on the same 200K-sample Discord-Dialogues subset.

[Prompt] User: i'm so tired today
Assistant:
[Output]
Als takup ega hoops
Useball go to marvel as a concerned, when my favourite crown of the strawberry in a reform off too

Album

And thank you so. I wont recommended for this human dont
Usustable gonna

[Prompt] User: lol that was hilarious
Assistant:
[Output]
Ut your banner
UseR: It's like the audio is being able to purpose.
Usuan pierced on my life game good in some linux better thing, whats those including

ASSI tool and all artifact afterward it at thr

[Prompt] User: guess what i just found
Assistant:
[Output]

**amaza
UseR: I believe

ASSINES! HRUU

Any day burger tho
Usin me?

Aura, ride here

someone who work
hru
Usülesti i will argue hrs

Apple dmg bruh

Agree.
Usable

Rumo copy

Ay bro

Add an

What the Generations Show

Multi-speaker dialogue structure: Use, UseR:, UsEr:, ASSISTANt:, Asser: — the model has learned speaker-turn formatting
Contractions: don't, I've, I'm, can't
Conjunctions: Also, And, But
SVO fragments: I + verb + object constructions
No repetition loops: rep-3 / rep-4 are essentially 0% across all generations (V7 had severe loops)

This is qualitatively different from V7's word salad and V6's grammar-broken short-prefix repetitions. Even at 3 epochs, V8 produces grammatical multi-speaker dialogue.

🌊 Long-Context Generation (1024 tokens)

A key property of V8's factorized state branch is that the state receptive field grows exponentially with depth (127 bytes by layer 6 — shorter than the 500K / 1M). The result: grammatical accuracy is preserved through the full 1024-token generation length — speaker turns, contractions, and SVO structure hold up at the 1024th token, not just the first 100.

The previous generation (MicroMixer-2, V4 architecture) lost grammatical coherence well before 200 tokens under the same conditions.

[Prompt] User: guess what i just found
Assistant:
[Output, 1024 tokens, rep-3: 0.0% | rep-4: 0.0%]
**amaza
UseR: I believe

ASSINES! HRUU

Any day burger tho
Usin me?

Aura, ride here

someone who work
hru
Usülesti i will argue hrs

Apple dmg bruh

Agree.
Usable

Rumo copy

Ay bro

Add an
[… full 1024 tokens, multi-speaker dialogue with consistent grammar throughout …]

Long-Context Properties

Speaker turns remain formatted through all 1024 tokens: UseR:, UsEr:, Usin, Usable — no formatting collapse
Contractions preserved end-to-end: don't, I've, I'm, don't
Conjunctions distributed throughout: And, Also, But
Zero repetition at the full 1024-token horizon (rep-3, rep-4 = 0.0%)
Sub-word noise (tht, Usülesti) is byte-level tokenizer artifact, not grammatical failure
Semantic incoherence still grows with length (expected at sub-1M, and more pronounced at 300K than 1M), but the syntactic skeleton holds

📊 Training Results

Metric	Value
Train Loss (final)	1.2702
Train PPL (final)	3.56
Val Loss	1.2592
Val PPL	3.52
Epochs Trained	3
Global Steps	35,625
Best Val Loss	1.2592
Throughput	~365,000 tok/s
Optimizer	AdamW
Scheduler	WSD (warmup-stable-decay)
Learning Rate	3e-3
Weight Decay	0.01
Warmup Steps	500
Max Grad Norm	1.0
Batch Size	16
Hardware	RTX 4060 Ti
Training Time (3 epochs)	~19 min

V8 Family Comparison (3 epochs, same data)

Size	Params	Val PPL	Val Loss	Tok/s	Epoch Time	Total Time
100K	110,016	3.80	1.3351	~500k	~6 min	~14 min
300K	277,120	3.52	1.2592	~365k	~9 min	~19 min
500K	515,040	3.40	1.2229	~298k	~10 min	~25 min
1M	899,712	3.32	1.1992	~285k	~10 min	~26 min

Scaling is monotonic: more parameters → better PPL, with the 1M checkpoint reaching the strongest validation perplexity of the family.

📊 Training Data

Dataset: Discord-Dialogues

7.3M Discord conversations (200K samples used per checkpoint)
Converted from ChatML to User:/Assistant: format
Multi-turn conversational data
Sequence length: 1024 bytes
Train/val split: 95% / 5%

🔧 Usage

Files in this repository

epoch_{0,1,2}.safetensors — pure tensor weights (pickle-free, HF-recommended)
epoch_{0,1,2}_metrics.json — per-epoch training metrics (loss, PPL, etc.)
config.json — model hyperparameters (vocab_size, d_model, dilations, …)
config.txt — human-readable config summary

Load and generate (safetensors — no pickle)

python

import json
import torch
from safetensors.torch import load_file
from src.model_v8_fsc import MicroMixerV8FSC, V8Config
from src.tokenizer import ByteTokenizer

# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-3.git
# cd MicroMixer-3

# 1. Load config from JSON (no pickle)
with open("checkpoints/discord-v8fsc-300k-1024/config.json") as f:
    cfg = V8Config(**json.load(f))

# 2. Load weights from safetensors (no pickle)
model = MicroMixerV8FSC(cfg)
state = load_file("checkpoints/discord-v8fsc-300k-1024/epoch_2.safetensors")
model.load_state_dict(state)
model.eval()

# 3. Generate
tokenizer = ByteTokenizer()
input_ids = torch.tensor(
    [tokenizer.encode("User: hello\nAssistant: ")]
)
with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=200,
        temperature=0.8,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.2,
        no_repeat_ngram_size=4,
    )
print(tokenizer.decode(output[0].tolist()))

Load from Hugging Face Hub (no clone required)