You can deploy this model and start earning money today!

changcheng967/
cpuflow-v97-memory

Name: cpuflow-v97-memory
Availability: InStock

Views

No views yet

CPUFlow v9.7 — Memory-Enhanced Semi-Coherent Model

Best semi-coherent model in the CPUFlow series. Adds RAM-Net sparse memory to the v5-LN cumsum backbone for a 1.7 PPL improvement without breaking coherence.

Results

Metric	v5-LN (baseline)	v9.7 (memory-enhanced)
Val PPL	11.94	10.23
Parameters	2.0M	2.47M
Speed	7,833 tok/s	3,369 tok/s
Coherent?	Semi	Semi
NaN events	0	0

Architecture

embed + CumStepPos → [RAMScanBlock × 6] → LayerNorm → tied output + FSP

RAMScanBlock:
  # Cumsum backbone (same as v5-LN)
  x_n = LayerNorm(x)
  h = W_proj(x_n)            # fused: d → 3k
  query, key, value = chunk(h, 3)
  key = sigmoid(key); value = tanh(value)
  scan_out = W_m(query * cumsum(key*value) / cumsum(key))

  # RAM-Net sparse memory sidepath
  addr = W_addr(x_n) → Product Softmax → Top-8 of 512 virtual slots
  mem_out = sparse_read_write(addr, x_n)
  merged = scan_out + W_mem_proj(mem_out)    # direct addition, no gate

  x = x + W_out(merged)
  x = x + ff_down(relu(ff_up(LayerNorm(x))))

Generation Samples

Prompt: "Lily and Tim went to the park. They"

...They saw many kids playing near the back house. They went up to a tree and gave them to their dad. They were very happy. After a while, they saw a big pile of ants. It was not a normal day. They did not want to play hide behind. Tim and his friends were scared, but they did not want to go home.

Prompt: "There was a little girl named Lily. She loved to play with her friends. One day"

...she put her shoes in the park. In the park, Lily saw a big lock on the ground. She wanted to open it. She tried to open the key, but it was too small. She tried to unlock the door open, but she could not.

Limitations

Semi-coherent at best. Named characters and pronoun tracking work early, but coherence breaks down ~100 tokens in.
"She tried to open the key" — semantic confusion from cumsum state blending.
Story drifts between scenes with no transition (park → church).
2.3x slower than v5-LN baseline due to memory overhead.
Trained on TinyStories only — children's vocabulary, no general knowledge.

Key Finding

Sparse memory (RAM-Net Product Softmax, 512 slots, Top-8) improves PPL by 1.7 points as a parameter-efficient capacity expansion. It does NOT solve entity tracking — at 2.5M params, the binding threshold (~160M params) makes entity-specific addressing impossible. The memory just adds raw capacity.

Usage

python

import torch
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
checkpoint = torch.load("best.pt", map_location="cpu")
# Build model (see train_cpuflow_v97_simple_memory.py for full architecture)
# Generate with temperature=0.8

See GitHub for full training code.

Citation

bibtex

@misc{Chang,
  title        = {FlashLM: CPU-Native Language Models Trained From Scratch on Free-Tier Hardware},
  author       = {Chang, Cheng},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.20113960}
}

MIT License.

changcheng967/cpuflow-v97-memory