A prototype model that is assembled as a mirrored transformer architecture with nested gating (adds an extra weight to the FFN) and morphological position encoding. It proposes that the model architecture creates different scaffolding, leading to different training regimens and capabilities.
Prisma is only viable as it piggybacks on pre-trained tokenizers and their weight-tied embeddings, it decomposes the transformer architecture into symmetric expand and compress phases that share structural weights, connected by a small number of unique middle layers. Information expands from tokens to semantics, then compresses back — like light through a prism.
architecture
Key Concepts
Mirrored layers. Each expand layer shares W1 (projection) and W2 (output) weights with its corresponding compress layer. The architecture gets 2N virtual layers of processing from N unique parameter sets. At 357M parameters, Prisma runs 41 virtual layers from ~20 unique weight sets + 1 middle layer.
G²LU — Gated-Gated Linear Unit. The gate is itself gated:
Where typical gated transformers have y = W2 @ (W1 @ x * silu(W3 @ x)), Prisma has:
python
1g4 = silu(W4 @ x)# inner gate2g3 = silu(W3 @ x * g4)# outer gate, modulated by inner3y = W2 @ (W1 @ x * g3)# gated output
One gate in function of the other. Creates quadratic (saddle-surface) decision boundaries instead of linear hyperplanes — each neuron computes a conjunction ("feature A AND feature B") rather than a single threshold. This produces narrow, separated activation channels that resist memorization and tolerate significantly higher learning rates. Part of the parameters saved with mirroring are re-distributed as W4.
WoRPE — Word-position Rotary Position Embedding. Dedicates a small subspace of each attention head to encode position within a word (0 = prefix, 1 = second subword, ...). The information is already in the BPE tokenizer's word-boundary markers — WoRPE surfaces it geometrically so the model doesn't have to rediscover it. No new tokenizer required.
Auxiliary skip prediction. An optional second head predicts t+K tokens ahead, providing gradient signal that rewards structural representations over local memorization. At K=1, functions as a dual-supervision regularizer through an untied projection.
Results
~50M scale prototype (WikiText-103, 4 epochs)
Model
Params
LR
WikiText PPL
LAMBADA
Standard SwiGLU
51M
1e-4
4125
0.002
Prisma (G²LU)
47M
1e-4
2914
0.001
Prisma (G²LU + WoRPE)
51M
1e-2
921
0.082
Standard trained 10 epochs; Prisma (G²LU + WoRPE) shown at 1 epoch — the point is LR tolerance, not epoch-matched comparison.
The regularization stack (mirroring + G²LU + WoRPE) enables training at 100x the standard learning rate without instability.
~350M scale prototype — comparison with published models
Prisma 357M trained on ~30B tokens (OpenWebText 20% + FineWeb-Edu 10BT continued training), compared against published models at similar scale.
Model
Params
Train Data
ARC-C*
ARC-E*
BoolQ
HellaSwag*
LAMBADA
PIQA*
WikiText**
WinoGrande
GPT-2 medium
355M
40B
0.250
0.436
0.586
0.394
0.430
0.664
26.75
0.531
Baguettotron
321M
200B
0.302
0.506
0.589
0.354
0.294
0.618
30.93
0.530
SmolLM-360M
360M
600B
0.359
0.640
0.550
0.536
0.455
0.715
19.49
0.570
SmolLM2-360M
360M
4000B
0.381
0.681
0.617
0.431
0.532
0.718
15.67
0.586
LFM2-350M
350M
10000B
0.393
0.662
0.642
0.489
0.399
0.698
25.68
0.558
Prisma
357M
30B
0.290
0.548
0.620
0.427
0.362
0.670
27.40
0.506
* normalized accuracy · ** word perplexity
training_budget
bench_vs_model
Key findings:
Beats GPT-2 medium on 5/8 benchmarks (ARC-C, ARC-E, BoolQ, HellaSwag, PIQA) with 25% less training data.
Beats Baguettotron (200B) on 6/8 benchmarks — including PPL — with 7x less data.
BoolQ 0.620 exceeds all models except LFM2 (10000B) and SmolLM2 (4000B). The anti-memorization properties of G²LU force genuine comprehension instead of statistical shortcuts.
ARC-Easy 0.548 — the largest absolute gain over GPT-2 medium (+11.2pp). FineWeb-Edu knowledge absorbed efficiently through G²LU's relational features.
Prisma wins on reasoning benchmarks (ARC, HellaSwag, PIQA, BoolQ). Models trained on 20-300x more data win on content prediction (LAMBADA, PPL). The architecture trades raw memorization for data-efficient knowledge application.
Disable G²LU, use standard SwiGLU in mirrored arch
Training
Flag
Default
Description
--lr
3e-4
Peak learning rate
--min-lr
0.0
LR floor for cosine schedule
--warmup-steps
100
LR warmup steps
--epochs
10
Training epochs
--batch-size
32
Micro-batch size
--grad-accum
1
Gradient accumulation steps
--context-length
512
Sequence length
--bf16 / --fp16
—
Mixed precision
--compile
—
torch.compile the model
Data
Flag
Default
Description
--data
—
Path or hf:dataset_name
--text-column
text
Column name for HF datasets
--tokenizer
gpt2
Tokenizer name or path
--num-samples
—
Limit dataset size
Architecture Details
Full spectral and representation analysis with all plots: ANALYSIS.md
Why Mirroring Works
Mirroring only works due to the additional gate. W3 and W4 specialize to serve different roles despite sharing weights — spectral analysis confirms the gates swap their stable-rank profiles at the architectural midpoint. The order of mirror layers may be rearrangeable, as the gates adapt to whatever representations flow through them.
Why G²LU Works
Standard SwiGLU creates hyperplane decision boundaries — broad, overlapping activation regions. G²LU's nested gate creates saddle surfaces — narrow activation bands with isolation gaps (like a spectral comb filter). This has three effects:
Anti-memorization. The gate geometry cannot form sharp, input-specific activations. The model is forced toward broad, relational features.
Higher LR tolerance. Narrow activation bands leave headroom between features. Large gradient updates shift features within their bands without colliding.
Compositional detection. Each neuron natively computes conjunctions (A AND B), not just thresholds. Might be useful for morphology, syntax, and structural reasoning.
G²LU can be seen as occupying a point between standard GLU (fixed activation, fixed gate) and KAN (fully learned activations): the activation function is fixed (silu), but its effective shape adapts per-input through the nested gate.
Why WoRPE Works
BPE tokenizers already mark word boundaries (Ġ for GPT-2, ▁ for SentencePiece). WoRPE surfaces this information geometrically in a dedicated subspace of the rotary embedding, so the model gets word-internal position for free instead of rediscovering it from attention patterns. Requires G²LU to exploit effectively — the saddle surfaces compute morphological conjunctions ("position-0 AND prefix-pattern") that single gates cannot.
Why Everything Works Together
The optimization landscape of this architecture is substantially more complex than a standard transformer — shared weights must serve both directions, nested gates must coordinate, and the hourglass bottleneck constrains information flow. This appears to be only tractable when anchored by pre-trained, weight-tied embeddings that provide a stable coordinate system. The frozen embeddings give the model fixed reference geometry, allowing convergence despite the architectural complexity.
Prisma grew from interpretability research on layer grafting (writing in progress) in Llama 3.2, which suggests that one of the ways that transformers might self organize to process language can be seen as like a mirrored structure that expands from tokens to semantics, then compressing back — bringing the interpretive analogy of seeing it as a biconvex lens with fractures or polarizing filters within its body. If the two halves are symmetric structurally, they can share weights. The gate (fractures/polarizing filters) becomes the minimum surgical unit for changing behavior. A single weightset becomes insufficient due to shared weights, which brought the question of how to properly make two gates efficiently collaborate.
G²LU emerged from the observation that for a pair of gates to be expressive and atomic, one gate needs to be in function of the other.
WoRPE emerged from noticing, that tokenizers already carry word structure but positional encodings ignore it — providing hints to the model allows faster convergence during training.
The architecture is a processing engine that plugs into pretrained tokenizer embeddings. The tokenizer is load-bearing infrastructure — Prisma operates within a pre-existing coordinate system.
Developer Notes
This model is the outcome of a POC done by a single individual with limited resources, further investigation, training and tests are being slowly conducted as time and conditions allow.
The proposed architecture was only fully trained on top of facebook/MobileLLM-125M tokenizer and weight-tied embeddings. It might be the case that it doesn't work as expected on untied embeddings and it is highly likely that it is impossible to train a model with this architecture without a pre-trained tokenizer.
Different arrangements of the architecture (varying middle layer count, mirror depth, width) would likely produce different results. Only this setup — with 1 middle layer — was tested, as a validation of whether the architecture works at all. The extreme case was chosen deliberately: if the bottleneck configuration most prone to failure still produces competitive results, less constrained configurations should too.
Factorized dimensions for embeddings and an intermediate down proj before the output head were attempted, and nothing useful came out of it.
It is completely unknown if the architecture is beneficial for larger models (1B+) — observations suggests it might.
Training
Architecture:
41 layers
20 with shared W1 and W2
1 unique
1024 dims
16 GQA heads, 4 KV heads (4:1)
vocab size 32k
RoPE + WoRPE + G²LU
Pretraining tokens: 30B
Precision: bfloat16
Tokenizer/Embeddings: facebook/MobileLLM-125M
Hardware: 1 H100
Disclaimer
This model is developed as a research model and it hasn't been tested thoroughly regarding synthesis and coherence quality, as its size is somewhat limiting. Use it at your own risk.
