🧠 Prajna CRN — Cognitive Resonance Network adapter for Gemma 4 E2B
Open weights. A 6.7M-parameter trainable "cortex" injected into the hidden
states of a frozen Gemma 4 E2B. On its training-distribution text it cuts
perplexity from 106.85 → 6.02 (≈18× lower) versus the frozen base — using only
0.3% of the base's parameters, trained CPU-only on a Mac Mini M4.
Prajna is not a standalone model and not a general reasoning upgrade. It is a
parameter-efficient hidden-state correction network: a small module that reads the
base model's intermediate hidden states and adds a gated correction back into the
residual stream. The base stays frozen; only the CRN trains. This repo ships the
trained weights and the minimal loader.
What it does (honestly)
| Result | Value | Scope |
|---|
| In-distribution perplexity | 106.85 → 6.02 (≈18× lower) | held-out training corpus |
| Trainable params | 6,721,432 | 0.3% of the base |
| Out-of-distribution text (bpb) | worse (~3×) | generic text |
| MMLU / BoolQ / HellaSwag | at or below frozen base | standard benchmarks |
| Implicit-goal / car-wash (reworded) | 0/8 | no generalization |
Read this carefully: the ≈18× is a corpus-compression result, not general
intelligence. On text outside its training domain the same adapter increases loss,
and it does not perform implicit-goal reasoning (a reworded car-wash probe scored
0/8; the widely-circulated "76% of models fail" test is not passed). We publish
this openly because the architecture is the interesting part, and the honest
limitations are part of the experiment.
Why the architecture is the innovation
The CRN is a small module injected at intermediate hidden states (here: Gemma layers
7, 15, 23, 31) that computes a correction from four cooperating sub-modules:
- Resonance Attention — frequency-modulated attention over hidden states using a
learned spectral filter bank (interpretable "cognitive bands", top-k selected).
- Skill Composer — a library of low-rank composable skills (top-k selection).
- Reflective Loop — a latent-space self-correction operator.
- Episodic Memory — a small differentiable key-value memory with read/write
gates and temporal decay, giving a persistent context channel without retraining
the base.
A per-injection sigmoid gate (crn_mix, learned) controls correction strength.
Because corrections are additive and gated, each injection is ablatable — we
measured that disabling injection @layer 7 alone raises in-distribution ppl from 6.02
to 13.93, while @layer 31 barely matters. This makes the method interpretable in a
way LoRA/adapters are not.
Key properties
- Base model is fully frozen (
no_grad); zero gradient flow into it.
- Backbone-agnostic: operates on extracted hidden states.
- 0.3% trainable params; trains on CPU in ~22 h.
Files
| File | Purpose |
|---|
dpo_final.pt | ★ Trained CRN adapter (SFT + DPO), 6,721,432 params |
memory_dpo_final.json | Trained episodic-memory state (required at inference) |
crn_components.py | Minimal loader: PrajnaStudentMultiLayer + CRN modules |
Training pipeline and data generation are private; this is an open-weights
release. The loader above is sufficient to run inference.
Quick start
1import torch
2from crn_components import PrajnaStudentMultiLayer
3
4student = PrajnaStudentMultiLayer(
5 device="cpu", inject_every=8, max_length=96,
6 num_frequencies=8, top_k=2, num_skills=32, skill_rank=4,
7 num_corrections=8, mem_size=256, mem_dim=64,
8)
9ckpt = torch.load("dpo_final.pt", map_location="cpu", weights_only=False)
10student.load_state_dict(ckpt["crn"], strict=False)
11student.load_memory("memory_dpo_final.json") # required
12student.eval()
13tok = student.tok
14
15ids = tok("Explain why the sky is blue.\n", return_tensors="pt").input_ids
16with torch.no_grad():
17 out = student._collect_hidden(ids)
18 logits, _ = student._apply_crn(out, training=False)
19print(tok.decode(logits.argmax(-1).flatten()))
The base google/gemma-4-E2B is downloaded automatically from HuggingFace on first
load (~10 GB). Set HF_HOME to an external disk if space is tight.
Note: the wrapped model runs on CPU; MPS is not supported for this loader
(a Gemma-4 embedding allocation issue).
Training (summary)
| Stage | Steps | Loss |
|---|
| SFT | 2000 | 0.2262 |
| DPO | 500 | 1.9788 (chosen > rejected) |
Hardware: Mac Mini M4, 16 GB, CPU only.
Limitations & future work
- The released checkpoint is a domain specialist, not a general model. It
overfits its training corpus and degrades out-of-distribution.
- No implicit-goal / common-sense reasoning is demonstrated (reworded car-wash
probe: 0/8). The earlier "reasoning" framing was a keyword-match artifact in
evaluation and has been retracted.
- Active research directions (not yet demonstrated):
- Make the correction generalize beyond the training domain.
- Validate reasoning transfer on reworded/held-out prompts.
- Scale injections and memory; explore larger bases.
We are publishing this as a credible, reproducible experiment — a genuinely novel
parameter-efficient architecture with an honest account of where it works and where it
doesn't. Contributions and critiques welcome.
🔗 Reference & author
Cite / reference:
1@misc{prajna-crn-2026,
2 title = {Prajna: Cognitive Resonance Network — a parameter-efficient hidden-state
3 correction adapter for frozen LLMs},
4 author = {eulogik},
5 year = {2026},
6 publisher = {Hugging Face},
7 howpublished = {\url{https://huggingface.co/eulogik/prajna}},
8 note = {Open-weights release (training code private)}
9}
If you build on Prajna or reproduce the in-distribution perplexity result, a link back
to
huggingface.co/eulogik/prajna is
appreciated. Feedback and collaborations welcome via the repo.
License
Weights: Apache 2.0. Loader (crn_components.py): Apache 2.0. Training code: private.
-green){kind=icon}
-lightgrey){kind=icon}
