In plain terms: DaisyChain-Train lets you use old / spare machines to
train neural networks. The training runs through emulated GPU logic —
verified INT8 units (GUDA-style) that stand in for a GPU's math — so machines
without a modern GPU can still do the work. Chain several together and they
train one shared model as a cluster.
Before you rely on it, see what it can't do → Limitations.
Use the hardware you already have to train. Each machine runs the emulated
GPU logic (verified INT8 units — multiply / requantize / ReLU) to compute the
model, and DaisyChain pools the machines data-parallel: device selection,
capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard.
Two ways to run — Docker or Python.
⚠️ Read this first
DaisyChain-Train is for small models on spare hardware. It pools compute,
not memory (the model must fit on one machine), scaling is sublinear, and
it is not a substitute for a real GPU on large models. Full envelope in
docs/LIMITS.md — please read it before relying on it.
Feature list
Python cluster trainer (daisychain/)
Data-parallel training across mixed machines — each node trains its own
shard; gradients combine into the exact full-batch gradient, replicas stay
bit-identical.
Capacity-weighted sharding — faster machines automatically take a
bigger share of the batch.
Emulated GPU compute (verified INT8 units) — VerifiedLinear layers
run every forward multiply / requantize / ReLU through the bundled trained
units; cluster-wide unit-invocation counts printed by rank 0.
Bring your own model — any Task (build_model / sample / loss)
via DAISY_TASK; template in examples/my_task_template.py.
Plain-float alternative task — same cluster and pooling with ordinary
float math.
Live dashboard (daisychain-dashboard) — readiness banner, P2P
connectivity scan, pooled cores/RAM, per-node capacity plan, live loss.
SpikeWhale control panel (spikewhale_panel, localhost:8899) —
sliders for model size / training settings, any HF dataset you can access
(default streamed FineWeb-Edu), start/stop/re-adjust, live loss.
Docker demo cluster — 3 nodes + dashboard in one command.
Windows helper (scripts\setup.bat) and Tailscale mesh guide.
DaisyChain-Web (web/) — browser P2P training
Zero-install nodes — opening the page IS joining; devices on one
network auto-group (Snapdrop-style, by public IP).
Private cross-network rooms — ?room=CODE with host approval for
every join.
Full WebRTC mesh — gradients travel peer-to-peer; the server only
signals and serves static files, it never sees weights or gradients.
Leader-follower runs — whoever presses Start sets width / sequence /
batch-per-device / steps / learning rate for the whole group; config
broadcast on the wire.
Mid-run join — late devices are synced in (weights + step) and
contribute from the next step.
Bit-identical replicas — same seeded init, strict roster-order gradient
averaging, deterministic Adam with identical state on every peer; verified
live by per-step weight hashes.
Sync guard — any weight-hash mismatch stops the run instead of
training past a fork; the step roster forbids silent partial averages.
Gradient repair — a follower missing a roster gradient re-requests it
from the leader (8 steps retained), bit-exact, and the run continues.
Cross-device kernel probe — every step, every device re-hashes a fixed
seeded int8 GEMM through its live kernel; catches broken arithmetic that
weight hashes cannot see.
Hardcoded FineWeb-Edu streaming — the server reads random slices of
the 10BT parquet shards straight off the HF CDN via HTTP range requests
(pure-JS hyparquet); built-in corpus fallback offline.
Checkpoints — download .pt, upload → broadcast to the whole
group; validated (magic, dims, tokenizer vocab) before accepting.
Inference kit — one self-contained HTML file with the trained weights
baked in; generations offline, anywhere.
In-page generation — prompt box on the trained model.
Old-hardware tier — no WebGPU? The identical units run on CPU (same
bits, so CPU and GPU devices co-train in one group). There is no
plain-float path.
Large-message fragmentation — multi-MB gradients/checkpoints chunked
at 48 KB over the data channels.
Verified compute & kernels (web)
Verified INT8 units everywhere — block-scaled int8 GEMM: exact LUT
products, exact int32 accumulation, bit-exact f32 epilogue with a pinned
rounding schedule; scales derived in JS f64 (division never runs on GPU).
Backends, best-first — DP4A hardware int8 dot → LUT compute shader →
CPU mirror; every kernel exact-gated at init (bit-level compares) and
demoted to the mirror on any mismatch.
Continuous random-cell audit at live training shapes.
Fused attention kernels — gather/scatter head-strided q·kᵀ and a·v
straight from BT×C layout (CUTLASS ex. 36/52 style).
QKV dual-GEMM fusion — shared left operand quantized once, one batch-3
dispatch (ex. 45); bit-identical.
B2B MLP chain — both MLP GEMMs back-to-back on GPU with fused per-row
absmax reduction and on-device quantize (ex. 13 + 23); WGSL-exact respec
with a fround-stepped JS mirror; fma-contraction-immune by construction.
Exact init gates on every kernel, every device, every boot — including
gates that "gate the gate" with discriminating boundary inputs.
IEEE-754 binary32 oracle in exact BigInt arithmetic — proves the JS
epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs).
Metamorphic property suite — reference-free relations + definitional
absolutes; 4/4 on an externally-authored bug corpus, matching the
differential gate.
RDNA2 ISA audit hardenings — bit-level (−0-aware) gate comparisons;
proof that FMA contraction cannot change the quantize.
Eleven-suite test chain (cd web && npm test) — convergence, replicas,
oracle, gates, properties, external corpus, self-corpus (the instruments
scored against my own bugs), B2B, optimizer, transformer LM, int8 backward;
results in web/TEST_RESULTS.md.
Dirty-buffer gate — the pool is poisoned before a re-sweep so state bugs
(a kernel assuming zeroed memory) are caught deterministically rather than
by ordering luck.
1docker compose -f docker/docker-compose.yml up --build
2# open http://localhost:8080
Brings up a 3-node demo cluster + dashboard on one machine.
Python (real machines)
On every machine (pip install -e .):
bash
1exportMASTER_ADDR=100.101.102.10 # coordinator IP (Tailscale 100.x recommended)2exportMASTER_PORT=295603exportWORLD_SIZE=34exportRANK=0# 1, 2, ... on the others5exportGLOO_SOCKET_IFNAME=tailscale0 # your mesh / LAN NIC6daisychain-train
Windows helper
scripts\setup.bat
An interactive menu: Docker, Python node, or just install deps.
🐋 SpikeWhale control panel (sliders → real training)
bash
1python -m daisychain.spikewhale_panel
2# open http://localhost:8899
A web control panel: pick model size / training settings with sliders, choose any
HuggingFace dataset you have access to (default: streamed FineWeb-Edu), hit
Start, and watch the live loss. Stop and re-adjust any time with
← Back to settings. Launches the real DaisyChain training underneath.
🌐 DaisyChain-Web (train by opening a browser tab)
bash
1cd web &&npminstall&&node server.js
2# open http://localhost:8787 on every device
Zero-install browser training: devices on the same network auto-group
(Snapdrop-style) and train a shared model peer-to-peer over WebRTC, computing
through the same verified INT8 units (WebGPU, with the identical units on CPU
for machines without it — there is no plain-float path). Private cross-network
rooms via ?room=CODE with host approval — the room creator accepts each
device before it can join. Includes gradient averaging with a deterministic
Adam optimizer (identical state on every peer, nothing extra over the wire),
checkpoint download (.pt) and upload → broadcast so one device can
restore the whole group after a failure.
Verification stack — the browser trainer's correctness is now checked by
things that run, not argued (full results):
IEEE-754 oracle (web/test_ieee.js): a binary32 oracle built from the
standard in exact BigInt arithmetic proves the JS epilogue mirror is
spec-correct — and rejects the old round-once mirror on 34% of inputs.
Metamorphic properties + oracle mutation scoring (test_metamorphic.js,
test_corpus.js): properties needing no reference implementation, scored
against an externally-authored bug taxonomy — 4/4, matching the exact
differential gate's 4/4. Relations own the loop bugs; two definitional
absolutes (ReLU output range, a unit-scale integer anchor) own the value
bugs no relation can see.
Exact kernel gates on every live kernel, a continuous audit at live
shapes, and a cross-device kernel probe (same seeded int8 GEMM, same hash on
every honest device, any backend). The audit's sampling was rebuilt against
a named bug class: its old constants (6 cells, 2% of GEMMs) bounded an
overhead that had never been measured (auditing every GEMM costs <0.01% of
a step), and uniform random cells cannot see a last-row/column bug at a
16512-wide output. Sampling is now stratified — the first cells are the
structural danger points, chosen deliberately. On a last-column bug, same
cell budget: uniform caught 5/300 audits, stratified 300/300, with
zero false positives.
RDNA2 ISA audit: reading a real GPU's shader ISA against our
determinism assumptions confirmed three of them on silicon (exact packed
int8 dot; correctly-rounded f32 add/mul; 1-ULP reciprocal — division stays
off the GPU) and produced two hardenings. (1) Real ISAs have non-IEEE
variants that flush −0 to +0; JS !== can't see that (-0 !== 0 is
false), so all gates and audits now compare bit patterns — exactly what
the replica hash sees. (2) FMA contraction of the quantize's x·inv + 0.5
(one rounding instead of two) turned out to be floor-invisible by
construction — proven in test_b2b.js with 175k+ last-ulp anomalies at
binade edges, zero surviving floor(). Rounding mode and denorm flushing
are runtime driver state on real hardware, which is why every device
re-runs the exact gates at every init.
Training data — FineWeb-Edu (10BT sample) is the hardcoded dataset. The
Space reads random slices of the parquet shards straight off the HF CDN with
range requests (pure-JS hyparquet, SNAPPY) and serves plain text at /data
— no dependency on the datasets-server rows API and its 503s.
Resilience — the sync guard now repairs instead of halting: a roster
gradient that reached the leader but not some follower (asymmetric WebRTC
mesh) is re-requested from the leader, bit-exact, and the run continues. The
guard still stops anything that would fork the weights.
CUTLASS-style kernel work, each step proven bit-identical or exact-gated:
Dispatch-optimized backward (ex. 05/24): independent GEMMs overlapped,
sibling trios fused into batch-3 dispatches — bit-identical gradients,
dormant int8-backward path down from 1.63× to 1.21× vs float.
QKV dual-GEMM fusion (ex. 45): q/k/v share one left operand — quantized
once, one batched dispatch, zero changed bits.
B2B MLP chain (ex. 13 + 23): both MLP GEMMs back-to-back on the GPU with
a fused per-row absmax reduction; the intermediate is quantized on-device
via a WGSL-exact respec (floor(f32(x·invScale)+0.5) — no GPU division)
whose fround-stepped JS mirror keeps mixed GPU/CPU fleets bit-identical.
Profile-driven speed work — every change below is bit-identical (gradient
and loss hashes unchanged), so none of it trades correctness for wall clock:
Buffer pooling: GPU buffers are recycled by size bucket instead of being
created and destroyed per dispatch (~19 per MLP call, per layer, per step).
6–10% faster, every hash unchanged.
Shared-operand embedding GEMMs: profiling put two f32 backward GEMMs at
55% of the entire step, and both consumed the same dlogits operand —
~17 MB at the 16512-token vocab, uploaded twice. One upload, one encoder,
one submit: that pair went 205 → 90 ms and the step 12% faster. The
fusion is gated bit-for-bit against the two calls it replaces.
A negative result, kept on purpose: the remaining hot kernel looked
cache-hostile (adjacent lanes wrote 66 KB apart), but making the writes
contiguous changed nothing. Two probes explain why — holding the output at
17 MB while cutting compute 32× barely moved the time. The logits GEMM is
transfer-bound, not compute-bound, and the readback cannot be removed
because softmax must stay in JS (WGSL's exp is not correctly rounded, and
a per-vendor exp would fork replicas). The real lever there is the
vocabulary, not the kernel.
An init backend race was tried and removed: it tied on the shipped path,
cost ~430 ms of init, and made the backend vary between page loads, which
silently invalidated three A/B comparisons before it was caught. A knob that
changes what you are measuring is worse than a fixed choice.
Dirty-buffer gate — and the assumption it falsified. Pooling introduced a
bug class the gates predate: a pooled buffer is not zero-initialized, so a
kernel that assumes zeros is right on step one and wrong on step two. That is
a state bug, where no single call is wrong and the sequence is, which is
the family no oracle can reach. The assumption was that the gates were blind
to it. Mutation-testing the gate proved otherwise: deleting the zeroing made
the plain gate fail at its second shape, because the sweep's own shapes
recycle each other's buffers. The suite had incidental coverage nobody
designed, which is coverage nobody can rely on — shorten the shape list and it
evaporates with the gate still green. It is now deliberate: the pool is
poisoned with 1e4-magnitude residue before a re-sweep, so detection no longer
depends on ordering luck. ~90 ms one-time.
Scoring the oracles against my OWN bugs (web/test_selfcorpus.js) — the
external corpus measures kernel bugs someone else wrote down, so this suite
asks the harder question: what do the instruments score against the four real
bugs of the month? Properties 0/2 on the data-plane pair (the c·out
theorem again), differential 2/2 — but half the bugs were not in the
kernels at all. A dead gate is a bug in a checker, caught only by mutating
the gate; a stalled roster gradient is a bug in the protocol, where every
computed value on every peer was correct, so no data oracle could fire. Those
needed different instruments, not better oracles.
All eleven test suites (cd web && npm test) pass; results with methodology in
web/TEST_RESULTS.md.
How it works
Each machine runs the same command; they form a cluster and train one shared
model. Two things happen:
The compute runs through the emulated GPU logic. By default the model is
built from VerifiedLinear layers, so every forward multiply / requantize /
ReLU is done by the bundled verified INT8 units (daisychain/verified/) —
the emulated GPU math. Rank 0 prints cluster-wide unit-invocation counts
so you can see the emulated logic doing the work.
The machines are pooled data-parallel. Each node trains on its own shard;
gradients are capacity-weighted and combined into the exact full-batch
gradient, so replicas stay bit-identical. Faster machines automatically
take a bigger share.
old machine A ─┐
old machine B ─┼─► each runs the emulated GPU logic on its shard ─► one model
old machine C ─┘ (gradients combined across the cluster)
Bring your own model
DaisyChain-Train trains any Task (build_model / sample / loss). Copy
examples/my_task_template.py, set DAISY_TASK=your_module:YourTask. Use
VerifiedLinear (see daisychain/verified_task.py) to run your model's compute
through the emulated units. See docs/CUSTOM_TASK.md.
Plain-float alternative
To skip the emulated units and train with normal float math on each machine, set
DAISY_TASK=daisychain.example_task:ExampleTask. Same cluster, same pooling —
the model math just runs as ordinary float instead of through the verified units.
The dashboard
daisychain-dashboard (or the Docker service) serves a Tailwind page at :8080
— readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan
(per-node device, weight, batch), and live training loss.
Networking
Use Tailscale for a P2P mesh so machines on different networks get stable IPs
on one interface — docs/TAILSCALE.md.
Layout
daisychain/cluster.py capacity-weighted CPU/GPU data-parallel trainer
daisychain/train.py entry point (daisychain-train)
daisychain/verified/ bundled trained N/N units + VerifiedLinear (train through them)
daisychain/verified_task.py default task: forward runs on the verified units
daisychain/example_task.py plain-float alternative task
daisychain/task.py the Task interface + loader
daisychain/dashboard/ agent + P2P scanner + Tailwind server
docker/ Dockerfile, dashboard image, compose (demo cluster)
scripts/setup.bat / setup.sh interactive setup helpers
config/ nodes + cluster env examples
examples/my_task_template.py starting point for your own model
test_verified_units.py full-domain checks of every shipped verified path (24)
docs/ QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE
daisychain/spikewhale_task.py trains the real SpikeWhale on streamed HF datasets
daisychain/spikewhale_panel.py slider control panel (localhost:8899)
web/ DaisyChain-Web: P2P browser training (WebRTC + WebGPU)
export_luts_web.py regenerates web/public LUTs from the trained units
Three changes to daisychain/verified/, each measured rather than asserted.
python test_verified_units.py (24 checks) reproduces all of it.
Bounded GEMM memory.LUTBackend.gemm and NeuralBackend.gemm materialized
the whole (m, n, k) product block before reducing it, so peak memory was cubic
in layer width. At the SpikeWhale panel's maximum (hidden 768, sequence 512) a
single layer's forward allocated 2.4 GB — on the spare hardware this project
exists to use. Both now block the m axis. The contraction axis k is untouched,
so the sum and its order are unchanged and results stay bit-identical:
GEMM
before
after
64x64x64
2.3 MB
2.3 MB (unchanged path)
256x256x256
135.8 MB
68.9 MB
512x768x768 (panel max)
2426.9 MB
77.2 MB
neural backend, 64x96x96
519.2 MB
57.5 MB
Caps are class attributes (LUTBackend.max_block_bytes,
NeuralBackend.max_products), tunable per instance. Below the cap the LUT path
takes the original single-block branch unchanged.
The two backends need different accounting, which is worth knowing before tuning
them: a LUT product costs 8 bytes, but a neural product is split into four nibble
pairs through a 128-wide net and costs ~880 bytes. Applying the LUT's byte budget
to the neural path computes a block 110x too large and never splits at all.
The multiply LUT is certified, not trusted.build_luts() returned its tables
unchecked while the docs described a self-certify gate. Because the table is the
complete finite domain, certifying it is the exhaustive verification:
LUTBackend.__init__ now checks all 65536 entries against signed integer multiply
and raises on mismatch. Cost: 0.5 ms.
Full-domain tests for the paths that ship. The units' exhaustive verification
covers the scalar entry points (NeuralMul8.verify() walks mul()); production
runs the batched ones — mul_array, relu_array, requant_array — and the LUTs
built from them, which are different code. test_verified_units.py checks every
shipped path against golden integer arithmetic over its complete domain. All pass
today; nothing would have caught a regression before.
Two guards against silent no-ops.gemm_int8 enforces the int32 accumulator
bound (MAX_INT32_K = 131071) instead of asserting it in a comment, since layer
width is user-selectable and an overflow would produce wrong numbers rather than
raise. instrument.require(**minimums) fails when a verified unit was
under-invoked, and refuses to report at all while counting is disabled — a zero
from a probe that was switched off is not evidence that the units did not run.
License: MIT · Author: Dean Byrne (Quazim0t0) · Org: DaisyChainAI
Citation
bibtex
1@misc{byrne2026daisychain,
2 title = {DaisyChain-Train: An Old Hardware Training Pipeline},
3 author = {Byrne, Dean (Quazim0t0)},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}},
6 note = {Chain spare/old machines into a data-parallel training cluster}
7}
