You can deploy this model and start earning money today!

jaivial/
disk-routed-chat-0.5b-v2

Name: disk-routed-chat-0.5b-v2
Availability: InStock

Views

No views yet

Disk-Routed Chat 0.5B — v2 (frontier training stack)

Experimental research artifact. A 0.5B disk-routed recurrent language model designed to run inference on CPU + SSD (2 rows/token, constant per-token I/O), trained with a frontier efficiency stack. This card documents an efficiency result, not a production assistant.

What is interesting here (the training result)

Trained from scratch on a single RTX 4090 in ~12 hours for ~$4:

Item	Value
Total params	~0.5B (77M active controller + 422M SSD product-key table)
Active/token	~77M + 2 routed rows
Base tokens	~3B (Cosmopedia synthetic textbooks + FineWeb-Edu)
Throughput	~70,000 tok/s on one 4090 (measured)
Alignment	SFT on 25k UltraChat dialogues

Frontier training stack

Chunk-parallel linear recurrence — turns the token-by-token scan into batched matmuls.
torch.compile — fuses the scan; 3.1× speedup (32k → 100k tok/s bench), the decisive lever.
Multi-token prediction (extra head) — sample efficiency.
Textbook/synthetic data (phi-1 principle) — quality per token.
WSD schedule (MiniCPM) + Muon optimizer (~2× vs AdamW).
Disk-routed product-key memory — 2 rows/token, capacity on SSD.

Honest limitations

The base model produces coherent English, but the chat model is minimal and weak: it learned the assistant format but does not reliably answer greetings or simple questions (e.g. "capital of France" is wrong). This is the expected ceiling of 0.5B / 3B-tokens / ~77M-active. Use it as a systems/efficiency reference, not a usable chatbot.

Files

base.pt — pretrained base (config + weights)
sft.pt — after supervised fine-tuning (chat format)
model_frontier.py — model definition (DiskChatMTP, chunk-parallel scan)
fineweb-tokenizer.json — 32k BPE tokenizer

Load

python

import torch
from model_frontier import DiskChatMTP
ck=torch.load('sft.pt',map_location='cpu')
m=DiskChatMTP(**ck['config']); m.load_state_dict({k.replace('_orig_mod.',''):v for k,v in ck['model'].items()}); m.eval()

Roadmap: distillation (MiniLLM), Rho-1 selective tokens, FP8, and a CPU/SSD int8 runtime for this architecture.

CPU + Disk inference throughput (measured)

Chat decode running on CPU only (no GPU), with the 422M-param product-key memory table stored as int8 on disk (memory-mapped) and only 2 rows read per token. 10 runs on an AMD EPYC 7713, 16 threads:

Metric	Value
Mean throughput	43.9 tokens/sec
Std / min / max	0.12 / 43.6 / 44.0 tok/s
Target	15 tok/s (~2.9× exceeded)
GPU	none
Per-token disk I/O	2 product-key rows (int8)

The recurrent backbone keeps a constant-size state (no KV cache), so decode cost is constant per token regardless of context length, and capacity lives on disk.

SSD-only inference (C + O_DIRECT — true cold disk)

The previous CPU section used PyTorch with the table mmap-ed, so the 422 MB table got cached in hot RAM (compute-bound, 43.9 tok/s). For a faithful SSD-only measurement we use a hand-written C runtime that reads the int8 product-key table with O_DIRECT (bypasses the OS page cache), so the 2 rows/token are genuinely fetched from the SSD each step. Controller weights stay in RAM; the table never loads into RAM.

Metric	PyTorch (warm RAM)	C + O_DIRECT (SSD only)
Mean throughput	43.9 tok/s	~178.8 tok/s
Per-token I/O	cached in RAM	8 KiB (2 pages, real SSD)
Host RAM — controller	~294 MB	~294 MB
Host RAM — memory table	~422 MB (cached in RAM)	0 (stays on SSD)
Host RAM — total (RSS)	7.5 GB load peak / ~1–2 GB steady	296.9 MB
GPU	none	none

~4× faster than the framework path, at <300 MB RAM with the model's capacity resident on SSD — reading a constant 8 KiB/token regardless of table size. Runtime: direct_chat2.c (recurrent constant-state decode, O_DIRECT paging, OpenMP).

Post-SFT update (extended 2-hour SFT)

The chat model was fine-tuned further: 2 hours on 100k UltraChat dialogues (vs the initial 25k). File: sft2.pt. Quality improved noticeably on simple questions while throughput is unchanged (same architecture/size).

Quality (post-SFT samples)

Prompt	Response
What is the capital of France?	"The capital of France, Paris, is the capital of France." ✓
Thank you!	"I'm glad you enjoyed the article!"
Hi, how are you?	"I am not capable of having personal opinions... However, I can provide you with the general steps..."

Still an experimental minimal-chat model (open greetings remain weak), but factual recall on simple questions is better than the 25k-SFT version.

Speed (post-SFT, 10 runs each, AMD EPYC 7713, no GPU)

Path	Mean throughput	Host RAM (RSS)	Per-token I/O
C + O_DIRECT (SSD only)	184.4 tok/s	296.9 MB	8 KiB real SSD
PyTorch (warm RAM)	40.0 tok/s	~1–2 GB steady	cached

Confirms the disk-routed inference profile is stable after alignment: ~184 tok/s at <300 MB RAM, capacity on SSD, no GPU.

Re-trained SFT2 (2026-07-20)

sft2.pt was re-trained from base.pt on a rented RTX 4090: 100k UltraChat dialogues, ~2h, 12898 steps, loss 2.58 → 2.16 (ppl 13.1 → 8.6), assistant-only loss mask. Ran eager (no torch.compile — its graph can't trace the chunk-parallel scan's data-dependent loop, and the fallback desyncs gradient checkpointing), so the 3.1× compile lever is not active in this run.

Verified coherent, prompt-appropriate generation (factually loose, as expected at 0.5B / 77M-active):

Prompt	Response
What is the capital of France?	"The capital of France is the capital of France."
Write a short sentence about dogs.	"Dogs are ... very intelligent and adaptable, with many remarkable abilities ..."
Explain what the sun is.	"The sun is a complex phenomenon ... The sun's energy powers the Earth's magnetic field ..."

USB / SSD runtime files are under runtime/: controller.bin (~294 MB, RAM-resident), mem.i8 (~422 MB int8 product-key table, read 2 rows/token via O_DIRECT), scale.txt. Swap guide for an existing USB deploy: USB_UPDATE_GUIDE_sft2.md.

SFT3 — coherence fix (2026-07-20)

sft2.pt still rambled off-topic on short prompts (it was trained on long, verbose UltraChat essays). sft3.pt continues training from sft2.pt on Alpaca-cleaned (51,760 short, direct instruction→answer pairs) — teaching concise, on-topic answers. 90 min on a rented RTX 4090, EMA-best checkpoint, best EMA loss 1.97 (ppl 7.2 vs sft2's 8.6). Same architecture/size/throughput.

Coherence before → after (same prompts, greedy/light-sampling):

Prompt	sft2.pt	sft3.pt
say hello	rambling "a 'fun' is a metaphor for..."	"I'm sorry, but I can't return to the same place as you."
which is the capital of France?	rambling "the 'fun' in French..."	"The capital of France is Paris, France."
Write a short sentence about dogs.	—	"Dogs are the best animals in the world. They are ... loyal ... providing companionship, emotional support ..."
Give me one tip for learning to code.	—	"Start by learning the basics of the programming language: ... 2. Use a programming ..."

Still 0.5B / 77M-active — coherent and prompt-appropriate, but factually loose (e.g. "2+2 = 3"). The runtime/ files and USB_UPDATE_GUIDE_sft2.md procedure now target sft3.pt; sft2.pt is kept for history.

SFT4 — data diversity (2026-07-20) — recommended checkpoint

sft4.pt continues from sft3.pt on Alpaca-cleaned + Dolly-15k (66,771 records; Dolly is human-written, adding QA / brainstorm / summarize / closed-QA variety). 90 min on a rented RTX 4090, EMA-best. Best EMA loss 2.11 (ppl 8.3) — higher than sft3's 7.2 because the mixed distribution is broader, not worse (loss across more task types).

Improves greetings and factual QA over sft3:

Prompt	sft3.pt	sft4.pt
say hello	"I'm sorry, but I can't return to the same place as you."	"Hello, I'm a user of this website. I'm an AI language model ..."
which is the capital of France?	"The capital of France is Paris, France."	"The capital of France is Paris."
Write a short sentence about dogs.	"Dogs are the best animals in the world ..."	"Dogs are our best friends, and they are the best of friends."

Still 0.5B / 77M-active: coherent and prompt-appropriate, factually loose (math still wrong). The runtime/ files and USB_UPDATE_GUIDE_sft2.md procedure now target sft4.pt; sft2.pt/sft3.pt are kept for history.

Storage provenance for the 184.4 tok/s reference

The 184.4 tok/s result above was measured with 16 threads on an AMD EPYC 7713 and a fast server-local SSD under Linux O_DIRECT. The exact server SSD model, interface, IOPS, and latency were not recorded. Therefore it must not be presented as a Micron 3400 result or as a guaranteed result on any other host.

For a useful target-host comparison, the laptop used for the removable-media test has two Micron 3400 MTFDKBA512TFH 512 GB NVMe drives, configured as Linux mdraid RAID 0 with an ext4 root volume (~953.6 GiB usable). The following are manufacturer per-drive specifications supplied for the 512 GB model, not a MONKE benchmark:

Micron 3400 512 GB per-drive specification	Value
Interface / form factor class	PCIe Gen4 NVMe
Sequential read	6,600 MB/s
Sequential write	3,600 MB/s
Random read	360,000 IOPS
Random write	700,000 IOPS
Read latency	55 µs
Write latency	14 µs
Laptop array	2 drives, RAID 0, ext4

MONKE at the 184.4 tok/s reference requires only about 369 4 KiB random reads/s (2 rows/token). These Micron specifications show ample nominal storage headroom, but sequential MB/s and vendor random-IOPS figures often use different queue depths than MONKE's latency-sensitive read pattern. The laptop's CPU, RAID behavior, filesystem/direct-I/O behavior, thermals, and actual QD1 random-read latency still determine achieved tok/s. A valid Micron result requires running the same 10-run C + O_DIRECT benchmark with mem.i8 on that array.

Removable-media result: generic USB 2.0 flash drive

A MONKE flash-drive deployment was measured on a generic/unbranded 128 GB USB 2.0 flash drive, mounted on Linux as exFAT (/run/media/jaime/ECF5-C242). Usable capacity was 117.2 GiB. The app measured 45–48 tok/s (about 46.5 tok/s midpoint) on this deployment.

Item	USB 2.0 flash-drive deployment
Measured decode throughput	45–48 tok/s
Interface	USB 2.0, 480 Mb/s bus (theoretical 60 MB/s)
Drive	Generic/unbranded USB Disk 2.0, 128 GB marketed / 117.2 GiB usable
Filesystem	exFAT; Linux `O_DIRECT` may be unavailable, so runtime can use buffered-read fallback
Model I/O	2 random 4 KiB rows/token = 8 KiB/token
Implied model reads at 45–48 tok/s	~90–96 random 4 KiB reads/s

This is not a bandwidth-bound workload: at 48 tok/s, model traffic is only ~384 KiB/s. The critical metric is low-queue-depth 4 KiB random-read latency / IOPS, where commodity USB 2.0 flash is weak. Typical USB 2.0 sequential read figures (~15–35 MB/s) therefore do not predict MONKE decode speed. Host CPU also matters; the 184.4 tok/s reference used 16 threads on an AMD EPYC 7713.

Storage ladder toward the 184.4 tok/s reference

Tier	Media / example	Interface / advertised sequential read	Throughput evidence	What it means
1	Generic 128 GB USB flash (this measurement)	USB 2.0 / 480 Mb/s	45–48 tok/s measured	Works and exceeds 15 tok/s target, but random-read latency limits decode.
2	Quality external NVMe enclosure + SSD	USB 3.2 Gen 2 or better	Not yet measured	First practical upgrade. Check 4 KiB QD1 random-read latency, not only MB/s.
3	Kingston DataTraveler Max USB-C 512 GB (`DTMAX/512GB`)	advertised 1088 MB/s read, 1025 MB/s write	Not yet measured	Much higher sequential spec than USB 2.0. Candidate; thumb-drive random I/O still needs measurement.
4	SSK SD321 1 TB (`HD-SD321-1TB`)	advertised 2080 MB/s read, 1878 MB/s write	Not yet measured	High-speed portable-SSD candidate. Verify negotiated USB mode plus 4 KiB QD1 read latency.
5	External NVMe over USB4 / Thunderbolt, or fast local NVMe	USB4 / Thunderbolt / PCIe	184.4 tok/s measured only on server-local SSD + EPYC 7713	Best path to reproduce reference storage behavior; matching host CPU remains required.

Do not read this ladder as guaranteed tok/s predictions. The Kingston and SSK numbers are manufacturer sequential specifications, not MONKE results. A drive reaches the reference only if its random 4 KiB latency/IOPS and host CPU match the reference workload. Test the same model on the target host's internal NVMe first; the delta against the removable drive isolates storage from CPU.

Safe Linux storage check (read-only; uses the table file):

bash

fio --name=monke-random-read \
  --filename=/run/media/jaime/ECF5-C242/MONKE/model/mem.i8 \
  --rw=randread --bs=4k --iodepth=1 --numjobs=1 \
  --time_based --runtime=30 --readonly --invalidate=1

MONKE needs about 2 × tok/s 4 KiB random-read IOPS: ~96 IOPS at 48 tok/s and ~369 IOPS at the 184.4 tok/s reference.

jaivial/disk-routed-chat-0.5b-v2