ENSEMBLE — a training-free AI built from portable experts
No gradient. No epochs. No GPU. Turn any dataset into a compressed expert file in one pass, then let a Kuramoto-coupled brain synchronize experts into emergent answers. ENSEMBLE thinks when idle, grows a persistent central memory, and safely rewrites its own coupling. It is built on the Palimpseste hypervector substrate.
ENSEMBLE is not a transformer and is not trying to be one. It is an experiment in a different direction: a training-free, ultra-compact, instantly-updateable, composable associative memory that grows smarter over time without retraining. This README is precise about what it is, what it does well, and where it loses to a trained 1B transformer.
What's new in v0.2
Global shared BPE tokenizer — one tokenizer across all experts lifts the char-level quality ceiling. On TinyStories (150 KB excerpt, D=5000): next-token accuracy 94.5% → 98.5%, query latency 112ms → 28ms (4× faster), RAM 187MB → 89MB (2× less).
Persistent central brain memory — the brain now accumulates synthetic concepts (from continuous thinking) in a memory that survives save() / load(). The system grows indefinitely: add experts + let it think, no retraining.
Brain save/load — full brain state persistence.
75 tests passing (was 49).
What's new in v0.3 — structural generalization
Structural query encoding — decompose a question into a PATTERN + a SLOT via explicit templates (Expert.from_qa_pairs(..., patterns=["what is the capital of {country}"])). Questions sharing a pattern are encoded consistently, so the answer format generalizes to unseen slots.
Measured generalization lift: on capitals (train: france/germany/italy/japan/egypt; holdout: spain/portugal/greece/brazil/norway/india/mexico/kenya), the char-level baseline returns an empty answer 100% of the time on unseen slots; the structural encoder returns a non-empty answer 100% of the time. That is +100 points of graceful degradation on the unknown.
Honest scope: this guarantees a well-formed answer shape, not factual correctness for a fully unseen slot. Without semantic embeddings, "spain" and "france" are orthogonal at the HV level, so the system imitates structure rather than interpolating meaning. The README is explicit about this.
91 tests passing (was 75).
What's new in v0.4 — embedding expert (optional semantics)
Embedding expert — an optional plugin that brings real semantic similarity to the brain. Dense word vectors (fastText, or a local mini-embedding) are projected into HV space via a seeded random-projection-then-sign (the new projection.py primitive), so spain and portugal become similar hypervectors (cosine-preserving, JL-style). This is exactly the ingredient the structural encoder needed for analogy to find the right slot.
It's a pure plugin: EmbeddingExpert satisfies the same duck-typed contract as Expert (relevance / candidate_hv / answer / signature_hv), so it plugs into the Brain and couples with lexical experts via Kuramoto with zero changes to the brain. The training-free / numpy-only core stays pure; semantics is opt-in.
Two construction paths: from_corpus_local (PPMI+SVD mini-embedding, training-free, numpy only, seconds — lower quality) and from_fasttext (load pretrained .vec, best quality, one-time download).
Honest result: the embedding expert measurably brings semantic similarity (spain~portugal = 0.18 vs ~0 without), which improves routing. However, with the mini-embedding the factual-correctness on the capitals holdout remains ~0% — the slot is found but the answer content is still decoded char-level. The pipeline is ready for fastText to deliver real factual gains; that's the documented next step.
106 tests passing (was 91).
What's new in v0.5 — factual generalization via dense analogy
Dense analogy solving — EmbeddingExpert.learn_relation("capital_of", {france: paris, ...}) + solve_analogy("capital_of", "spain") implements the classic Mikolov analogy (paris − france + spain ≈ madrid) in normalized dense space, averaging the (answer − slot) direction over known examples and finding the nearest word — excluding the query slot so it doesn't parrot itself.
Wired into structural answers — when an embedding expert is attached to a lexical expert (from_qa_pairs(patterns=..., embedding=emb)), unseen slots are answered via dense analogy instead of char-level decode, returning the plain-text answer.
Measured factual gains (fastText, end-to-end via the Brain): on a capitals holdout (train: france/germany/italy/japan/egypt/russia; holdout: spain/portugal/greece/norway/china/india/brazil/turkey), factual correctness goes from 0% (char) / 0% (structural-only) to ~50–67% (structural + embedding analogy). Concretely: spain → madrid, portugal → lisbon, greece → athens, norway → oslo are recovered correctly despite never being in training.
Morphological filtering (default on): excludes candidates sharing the query slot's stem (filters indian, turkeys, brazilian), which empirically recovers india → delhi (50% → 62% on the isolated analogy benchmark). Two scoring methods exposed: cosadd (default, Mikolov 3CosAdd) and cosmul (Levy & Goldberg 3CosMul) — measured cosadd+morph to be the best on this benchmark.
Honest scope: fastText analogies are imperfect (china → porcelain on a miss) — that's fastText's distributional ceiling, not the architecture. More known pairs and larger vocab would lift it further. The point stands: the wall is broken.
109 tests passing (was 106).
What's new in v0.6 — empirical tuning + scale + concrete demo
k-NN local analogy — solve_analogy(k_neighbors=5) averages the relation direction over only the k nearest known slots to the query, instead of all known pairs. Measured empirically: k=5 is the sweet spot. Counter-intuitively, more known pairs hurts (12→26 pairs drops accuracy 62%→50%) because distant examples (European vs Asian capitals) dilute the direction. The local neighborhood keeps it coherent.
Scaling to a real corpus (TinyStories, 90 MB) — char-level saturates at 94% next-token accuracy from D=5 000; BPE reaches 99.3% at the same D while using 3× less RAM (117 MB vs 375 MB). The substrate scales.
Multi-domain demo (examples/multi_domain_demo.py) — a brain holding 3 lexical experts (geography/math/science) + the embedding expert, answering cross-domain questions and recovering 4/4 = 100% of unseen capitals (spain→madrid, portugal→lisbon, greece→athens, norway→oslo) in a single end-to-end run.
Honest limitations documented: the fastText analogy ceiling on the capitals holdout is ~62% on the full 8-country benchmark (china→porcelain etc. are real distributional false friends); cross-domain routing (Kuramoto dispatch between lexical experts) is not yet calibrated — the embedding expert can over-dominate because lexical signatures are quasi-orthogonal to queries. Both are real, measurable, and the next steps are clear.
109 tests passing.
The four ideas
1. Dataset → Expert (.exp) — one pass, no gradient, smaller than the source
Any dataset becomes a frozen, portable expert file in a single pass. There is no gradient descent and no epoch — learning is an O(1)-per-token write into an append-only memory. The expert file is smaller than the dataset (typically 3–9×): hypervectors are never stored; the symbolic token stream is gzipped and HVs are rebuilt on load from a deterministic encoder.
2. Kuramoto brain — experts couple and synchronize into emergent answers
Load several experts into a Brain. Each expert becomes a Kuramoto oscillator (natural frequency = relevance, coupling = signature similarity). The oscillators synchronize by similarity; the emergent attractor is a state no single expert produced. The brain routes questions to the right expert or composes answers when two experts know the topic.
3. Global shared BPE + persistent central memory — the brain grows
A single BPE tokenizer makes all experts token-compatible (so they share the same vocabulary and can be assembled freely). Meanwhile the brain's central memory accumulates synthetic concepts discovered during continuous thinking — and unlike a transformer's frozen weights, this memory persists and grows. Adding knowledge is Lego: drop in a .exp, let the brain think, and its concept store expands. No joint retraining.
4. Continuous thought + safe self-modification
When idle, the brain thinks: it samples remembered queries from one expert, asks all the others, runs the attractor, and writes novel coherent results into the central memory. It can also rewrite its own coupling under a Lyapunov constraint (ΔE ≤ 0). The acceptance rule is immutable — the recursion is bounded by construction.
How it differs from an LLM
Transformer LLM
ENSEMBLE
Learning
gradient descent, ~1T tokens
one-pass write, no gradient
Adding knowledge
retrain / fine-tune / RAG
drop in a .exp (Lego) + think
Model size vs data
grows with parameters
expert is smaller than its data
Grows over time
frozen weights
central memory accumulates concepts
Idle behavior
nothing
dreams across experts, writes concepts
Self-tuning
hyperparameters fixed
rewrites its own coupling (Lyapunov-bound)
Hardware
GPU
plain CPU
Quick start
This repo is self-contained: it vendors the palimseste substrate, so a fresh clone runs with only numpy.
bash
1git clone https://huggingface.co/thefinalboss/ensemble
2cd ensemble
3pip install numpy # the only runtime dependency4python -c "from ensemble import Brain, Expert; print('ok')"
Build experts and assemble a brain
python
1from ensemble import Expert, Brain
23# Option A: char-level (backward compatible, simplest)4math = Expert.from_qa_pairs(5[("what is pi","pi is approximately three point one four")]*5,6 domain="math", D=10000)78# Option B: global shared BPE (recommended — higher quality)9corpus =open("some_corpus.txt").read()10bpe = Expert.build_bpe(corpus, vocab_size=2000, D=10000)# train once11math = Expert.from_qa_pairs(12[("what is pi","pi is approximately three point one four")]*5,13 domain="math", D=10000, tokenizer=bpe)# share it14geo = Expert.from_qa_pairs(15[("what is the capital of france","the capital of france is paris")]*5,16 domain="geography", D=10000, tokenizer=bpe)1718brain = Brain()19brain.add_expert(math)20brain.add_expert(geo)2122print(brain.query("what is pi").answer)# -> math23print(brain.query("what is the capital of france").answer)# -> geography2425brain.think(seconds=10)# the brain dreams -> writes to central memory26brain.self_modify()# safely retunes its own coupling2728# the central memory persists29brain.save("mybrain")30brain2 = Brain.load("mybrain")# concepts survive reload31print(f"{brain2.n_concepts} concepts persisted")
Saving experts (compressed)
python
1result = math.save("math.exp")# source -> expert, compressed2math2 = Expert.load("math.exp", tokenizer=bpe)# BPE experts need the shared tokenizer
Structural generalization (unseen slots)
python
1# Train on capitals of some countries, WITH a template.2# The template lets the expert generalize the *answer format* to unseen slots.3geo = Expert.from_qa_pairs(4[("what is the capital of france","the capital of france is paris"),5("what is the capital of germany","the capital of germany is berlin")]*4,6 domain="geo", D=10000,7 patterns=["what is the capital of {country}"])89# spain was NEVER in training — yet the structural expert answers (format-generalized),10# where a char-level expert would return empty.11print(geo.answer("what is the capital of spain"))# non-empty, by analogy
Scaling to "1B-equivalent"
ENSEMBLE has no stored parameters, so "1B" means capacity — distinct associations the memory holds without collision, exponential in dimension D. The 1b preset uses D = 100 000.
BPE vs char-level on TinyStories (150 KB excerpt, D=5000)
metric
char
BPE
delta
next-token accuracy
94.5%
98.5%
+4.0 pts
query latency
112 ms
28 ms
4× faster
RAM
187 MB
89 MB
2× less
build time
46s
107s
one-shot cost
The global BPE lifts the quality ceiling exactly as predicted: higher accuracy, far lower latency (fewer, longer tokens), and half the RAM. The tradeoff is a slower one-time build.
ENSEMBLE scaling with D (facts corpus)
D
next-token acc
QA recall
.exp/source
2,000
30.6%
0%
7.0×
10,000
93.5%
35%
7.0×
100,000 (1b)
94.7%
40%
7.0×
Generalization to unseen slots (capitals benchmark)
Train on capitals of 5 countries; hold out 8 unseen countries. Metric: % of holdout returning a non-empty, well-formed answer.
mode
non-empty on unseen slots
char-level
0% (silent failure)
structural (patterns)
100% (graceful, format-generalized)
Honest: the structural answers are well-formed guesses by analogy, not factually correct for fully unseen slots (no semantic embeddings). The win is graceful degradation — the system says something sensible instead of nothing.
Honest verdict vs 1B transformers (TinyLlama-1.1B, Pythia-1B)
ENSEMBLE wins on: zero training (~30s CPU vs a GPU cluster on 1T tokens), footprint (a 1b expert is ~2 KB vs ~2 GB), instant knowledge injection, compositionality, near-perfect memorization of seen data (95–99% with BPE), and continuous growth (the central memory never stops accumulating).
A 1B transformer wins on: broad world knowledge (MMLU, HellaSwag), generalization (ENSEMBLE memorizes; holdout QA ≈ 0%), fluency on unseen text, reasoning.
Bottom line: ENSEMBLE is a different tool. For narrow domains with known data and a CPU-only / tiny-footprint constraint, it is competitive or superior. For general intelligence, it is not — yet.
Architecture
Expert dataset -> compressed .exp (one pass, no gradient)
└ build_bpe train a global shared BPE tokenizer
ExpertOscillator an expert as a Kuramoto oscillator
ExpertKuramotoAttractor multi-expert synchronization -> emergent attractor
Brain conductor: roster, query, think, self-modify, save/load
BrainMemory persistent central memory (grows via thinking)
ContinuousThinking cross-expert dreaming -> writes concepts to BrainMemory
SelfModifier Lyapunov-bounded retuning of coupling
What this is (and is not)
Is: a training-free associative memory built on hypervector (VSA) algebra; an expert system where knowledge is portable, compressed, composable, and grows over time; a research artifact exploring Kuramoto coupling + persistent concept memory. CPU-only, numpy-only, auditable.
Is not: a transformer; competitive on broad-knowledge benchmarks; a generalizer (it memorizes what it is shown).
