A living AI that thinks continuously, remembers forever, and grows on its own.
Fractus is NOT a transformer. It's a Continuous Cognitive Agent — a dynamical system that maintains a persistent thought state, advances it tick by tick through 16 blocks, and routes via Kuramoto oscillator phases. The checkpoint is a living seed: it never freezes, grows at runtime, and trains forever.
Status (2026-08-28): Training live on QuickPod 8×5090 (boost_v4, ~89M/430M on GPU0). Decode surgery I: window-64 + anti-copy — Space Retail×32 is dead, unique@32 now 12–27, still not English. Gate = unique@40 greedy PREFIX. See docs/2026-08-28-DECODE-WINDOW.md.
The .pt checkpoint contains the full model (weights + dynamic state). You need this repo's code to run it — Fractus is a custom architecture, not a transformer. Checkpoints are on HF.
License
Python
PyTorch
Params
Active
Status
Datasets
What is Fractus?
Fractus is not a chatbot. It's not GPT. It's not a transformer.
Fractus is a Continuous Cognitive Agent — an AI that works like a brain, not a calculator. Instead of processing input → output in one pass, Fractus ticks like a biological system: it maintains a persistent thought state, advances it through multiple blocks of processing, remembers everything across sessions, and can grow new capacity by itself.
What makes it different from GPT/Claude?
GPT-4 / Claude
Fractus
Thinking
One pass, done
Continuous ticks (like a heartbeat)
Memory
Forgets when context window fills
Remembers forever (survives restarts)
Learning
Retrain from scratch ($$$)
Learns from every interaction
Growth
Fixed size forever
Grows new experts at runtime
Mental states
One mode always
Shifts between cognitive modes
Where it runs
Corporate cloud
Your machine
Training
Fixed, done once
Perpetual, never stops
Architecture (1.05B total / ~119M active per token)
ContinuousThoughtEngine
├── d_model=1280, 16 layers, 16 heads
├── FractalLinearAttention multi-level causal linear attention (O(L)),
│ carry state (S, z) persists across chunks
├── Kuramoto phase clock RK4-integrated oscillators → phase vectors
│ (learned end-to-end; feeds routing)
├── PhaseRoutedMoE 128 experts, top-2 active per token,
│ von Mises gate over phases, Farey-sequence
│ expert phases, load-balance loss in objective
├── Low-rank experts W = scale · U@Vᵀ (rank 64) → 64× less compute
├── Tied embedding/head GPT-2 BPE vocab (50257)
└── Persistent thought state residual stream carried across ticks
Parameter accounting: ~1.05B total, but only ~118.8M are active per token (dense attention + top-2 of 128 sparse experts) — ~183.1M including the tied embedding. An 11.3% sparsity ratio. This is the basis of the adapted scaling target below.
The 12 Building Blocks
Block
What it does
Continuous Thought Engine
The brain — thinks tick by tick through 16 blocks
Persistent Memory
Vector bank surviving sessions, cosine recall, 5% blend injection, salience head gates impact
Cognitive Modes
Mental states discovered unsupervised (k-means on Kuramoto phase features): focused, creative, exploratory
Decides its own actions: retrieve, learn, generate
Progressive Growth
Grows from 6M to 1B+ params, palier by palier (maybe_grow: width + depth + experts)
Self-Modification
Adds new experts at runtime when routing is imbalanced (zero-init, placed near the dominant expert)
PhaseRoutedMoE
Sparse experts routed by oscillator phases
Kuramoto Clock
A dynamical system that drives routing decisions
Online Trainer
Learns continuously, one chunk at a time
Vision (tick_vec)
Multimodal input path — image patches drive the engine directly, bypassing the token embedding (CIFAR "eyes" prototype trained while the text run continued)
Live Training Status (28 August 2026)
Hardware: QuickPod 8× RTX 5090. One independent Python process per GPU. No DDP / no gradient sync. Consolidation = mean-merge of the 8 .pt when we need a single brain.
Resume point 2026-08-28 14:40 UTC after RunPod SSH death: GPU0 79,922,176 tokens (~18.6% of this pass). Other GPUs 76.2M–78.4M. Exact JSON: checkpoints/x8run/RESUME_gpu{i}.json
Trainer: scripts/fast4gpu_boost_v4.py
Config: BATCH=4, SEQ=128, LR 7e-4, SGD+momentum 0.9, bf16, TF32, compile off, FRACTUS_ATTN_IMPL=chunked, BLOCK_CKPT=1, SS_RATE=1.0, SS_PROB 0.2→0.5 over 50M tokens, REPEAT_COEF=0.1, P0 on, PROBE_EVERY=0 (live unique@40 probe crashed the first QuickPod launch — probes are offline)
Throughput after resume: ~840–940 tok/s/GPU. Remainder of this pass ≈ 4.1–4.5 days if 8 GPUs stay up
Torch on RTX 5090: image default 2.2.1+cu121 has no Blackwell kernels. Required: torch ≥ 2.11 + cu128
Crash recovery: hourly upload of checkpoints/x8run/fractus_1b_gpu{i}.pt + RESUME_gpu{i}.json. That folder is the source of truth. See docs/2026-08-28-QUICKPOD-RESUME.md.
Speech gate (not the loss):unique@40 greedy PREFIX — 40 generated tokens, count distinct ids, no ban, no temperature. Post window-surgery unique@32 (2026-08-28 23:35 UTC): 16 / 14 / 27 / 12. Mono-token lock broken. Short cycles remain. NO-GO for speech. Teacher-force CE going down does not mean Fractus speaks. Do not cite banned-decode diversity as speech.
Vision — the CIFAR "eyes" prototype
Fractus is not text-only. The engine exposes tick_vec(obs_vec): a multimodal entry point that accepts a precomputed (B, d_model) vector and injects it directly into the residual thought state — bypassing the token embedding entirely. Any modality that can be embedded into d_model dimensions can drive continuous thought: images, audio, sensor streams.
image → PatchEmbed → (B, d_model) patch vectors
│
▼
engine.tick_vec(patch) ← no tokenizer involved
│
thought state advances through all 16 blocks
(same Kuramoto routing + MoE as text)
Proof of concept — CIFAR-10 eyes: a small CTE + PatchEmbed stack (fractus/nn/vision.py) was trained on CPU on real CIFAR-10 images (fractus_eyes_cifar_final.pt), demonstrating that image patches can drive the continuous-thought loop. Crucially, this ran as a parallel track: the 8-GPU text run was never interrupted for eyes work — GPU text digestion and CPU vision learning proceed simultaneously on the same living system.
This mirrors the biological premise: eyes evolved as a peripheral sense feeding a central dynamical brain, not as a core feature of it. The text 1B is the brain; vision is a front-end that plugs into tick_vec.
Adapted Chinchilla target
Standard Chinchilla (20 tokens/param) assumes dense transformers where all parameters train on every token. Fractus violates that: only ~119M of 1.05B params are active per token, and the Kuramoto clock trains end-to-end now. The adapted target is 20 × active params ≈ 2.4–3.7B tokens; with a warm-started checkpoint, the practical target is ~2.5–3B tokens — which is why the phase-2 corpus is 3.44B. Since Fractus grows continuously (maybe_grow), this is an instantaneous cumulative-exposure requirement that grows with the model, not a final-stop condition. Details: docs/2026-08-12-fractus-chinchilla.md.
The Surgeries (mid-training interventions, no weight wipes)
A defining discovery of this run: the brain (.pt) and the code are separable. Bottlenecks were fixed by live surgery — save the checkpoint, patch the code, reload weights, resume at the exact recorded token offset. Multi-day digestion is never thrown away.
Phase
Intervention
Result
A — Initial
Last-position-only CE, 4 independent GPUs
Loss fell; generation collapsed into single-token loops
B — Routing surgery
Kuramoto was frozen under no_grad (order parameter r ≈ 0.01–0.03); LB loss was detached → ~70% of experts dead. Fixed: gradients enabled (state kept detached for carry), CE + 0.02·lb, gate temp 1.0 → 2.5, omega scale ×4
lb ≈ 14 live on all GPUs; experts alive
C — Dense CE
Replaced last-position CE (1 target / 128 tokens) with CE over all positions
Sharp loss drop; token-to-token chaining enforced
D — Decode surgery
Phase/thought noise, frequency penalties, cycle bans, forced escape tokens
Loop lock broken; still no coherent English
E — Train/gen mismatch
Training used causal attention + RK4 Kuramoto; generation used a simpler single-tick Euler path. Aligned decode via generate_aligned.py
Decode now matches the training path
F — Loss recalibration
Cumulative-average CE was misleading near 2.0 → batch CE + EMA; LR 1e-3 → 5e-4
Trustworthy metrics
G — Scheduled sampling
Two-pass training: TF CE + LB, plus SS steps mixing model samples into inputs
Live; now SS_RATE=1.0 on boost_v4
H — Pod death / exact resume
RunPod SSH died 2026-08-28. Reloaded 8× .pt + manifests on a new QuickPod 8×5090 at the recorded start_token_next. Torch upgraded 2.2→2.11+cu128 for Blackwell
Weights kept. Pass continues from ~80M/430M
I — Decode window + anti-copy
Length-1 tick_chunk + carry locked Space to Retail×32. Default decode is now causal window 64 + mask previous token. Weights / train loop unchanged
CE with ground-truth history + continuous internal state — is the model digesting data?
live
ema_ss
CE after scheduled sampling mixes model-generated tokens in — partial free-run robustness
live
AR
Warm on 32 true tokens, greedily free-run 32 steps, CE vs truth — actual generation quality
offline
A single CE number cannot represent both teacher-forced learning and free-run generation. Live ema_tf after the QuickPod resume is mid-teens on the lead GPUs (first steps inflate EMA — wait). AR stays the offline generation metric and last unique@40 was NO-GO. Random-vocab CE ≈ 10.82; AR in the thousands (random guessing over the GPT-2 vocab is ~10.82): free-running compounds every error, and teacher forcing always supplies the correct past. Trust ema_tf/ema_ss for learning progress, AR + text probes for generation progress. The convergence signal is AR falling toward the ss/tf order of magnitude, plus readable output. Details: docs/TRUSTED_LOSS.md.
Research Results (Honest)
Refuted:
EDT (Expert Decoupled Training) — all 5 variants ~19–20% worse than from-scratch; MSE objective misaligned with CE, router ignores half the experts.
Forward-Forward (Hinton 2022) — NLL rose from 124 to 221; local objectives can't replace global backprop here.
Validated:
Progressive growth — warm start converges faster; trained through palier 3 (350M, loss 23.0, ~5 days CPU).
Sparse low-rank MoE — 2/128 experts = 64× less compute.
Open-heart operability — live surgery on a training model works (see above).
Routing pathology as a first-class debug target — expert-hit histograms and the Kuramoto order parameter catch failures that loss curves hide (they caught the frozen clock and the dead experts).
Training optimizations (measured): tied head (~1.1×), head-partial training (~2×), sparse gathered low-rank MoE (8× at 16 experts, 64× at 128), detached-state Kuramoto, gradient accumulation (~1.4×), SGD+momentum over AdamW (~1.37×), batching (335 → 1345 tok/s at B=8), bf16 (~2×). Combined CPU: ~336× over naive. Measured on CPU: 707 tok/s single-stream, 1345 tok/s batched.
Literature, philosophy, occult, masonry, science, medicine
wordnet
3M
117K dictionary synset entries
Total
~4.2B
Tokenized streams: Phase 1 = ~1.52B tokens (.pt files); Phase 2 = ~3.44B tokens (8 GPT-2 BPE int32 memmap shards). Phases are kept separate to avoid re-ingesting the same ordered stream.
Applied Neuroscience — the theoretical core
Fractus is a neuroscience-grounded architecture: real brain mechanisms are mapped to software/AI patterns, and that mapping is itself training data. Every entry below is present in the dataset — verified by file listing, not just claimed.
Tick: one step of thinking. The engine processes an observation, updates its thought state through all blocks, and optionally emits output.
Thought state: a vector that persists across ticks — the engine's "consciousness."
Chunk: 32 tokens processed in one forward pass. The thought state and per-block attention state carry between chunks.
Expert: a small low-rank network (W = scale·U@Vᵀ) that specializes in certain thoughts. Only 2 of 128 active per token.
Kuramoto clock: coupled oscillators producing phase vectors that route tokens to experts — learned end-to-end since the routing surgery.
tick_vec: the multimodal tick — feed any precomputed (B, d_model) vector (image patches, embeddings from any encoder) straight into the thought state without tokens. This is how vision plugs in.
Surgery: patching code around a preserved checkpoint mid-training, resuming at the exact token offset. Weights are never wiped for a routing, objective, or decode bug.
Limitations (stated plainly)
Generation is not yet coherent English — word-level repetition loops / lexical noise. Exposure bias is being addressed by scheduled sampling; AR is the metric to watch.
GPT-2 vocab dominates parameters (81% at d=768) — vocab reduction is a known lever.
"Remembers forever" and "grows on its own" describe the architecture's design; no independent benchmarks are provided.
This is a research artifact and a live training run, not a production assistant.
License
MIT. Fractus belongs to you, not to a corporation.
