Hybrid SPLM (Scalar-Potential + Attention Language Model)
The Hybrid SPLM combines an attention front-end with a scalar-potential refinement back-end in a two-stage architecture. The attention blocks gather global context across positions (what attention does best), then the SPLM blocks refine each position deterministically through a learned energy field (what conservative dynamics does best). At decode time, the SPLM tail costs \(O(d^2)\) per token independent of sequence length -- the FLOP-efficiency hypothesis the hybrid is designed to test.
This model achieves 8.50 PPL on TinyStories, within 0.69 PPL of matched pure attention (7.81), and is the best-performing variant in the Semantic Simulation SPLM family.
The Hybrid SPLM (Variant A) is a two-stage autoregressive language model:
Attention stage (k=4 blocks): Standard causal multi-head self-attention blocks with residual connections, producing a contextualised representation.
SPLM stage (m=4 steps): Conservative scalar-potential dynamics that refine the attention output through gradient-driven integration steps.
The single causal cumulative-mean context \(\xi\) is re-derived from the detached attention output, preserving the causal-honesty invariant. A single shared \(V_\theta\) drives all integration steps.
Developed by: Dimitar P. Gueorguiev (Independent Researcher)
Model type: Hybrid attention + conservative autoregressive language model
Effective damping. The learned \(\gamma = 0.166\) overstates the true dissipation. The LayerNorm applied after each SPLM integration step rescales the hidden state, absorbing most of the velocity decay. The dynamics in the SPLM refinement tail are therefore heavily underdamped even at this nominal value. Gamma-sweep experiments on the OpenWebText-scale Fock-PARFLM variant confirm that the effective damping γeff is much smaller than the nominal coefficient across all SPLM-family models.
Key Design Properties
Best of both worlds: Attention handles global token routing; conservative dynamics handles local refinement.
FLOP-efficient decoding: At long context \(T \gg 1\), the SPLM tail adds \(O(d^2)\) per token vs \(O(Td)\) for attention with KV-cache. The embed+logits floor limits short-context savings to ~9%, but at \(T \geq 4096\) the hybrid achieves -39% decode FLOPs at PPL parity.
Helmholtz decomposition: The two-stage architecture realises a learned Helmholtz decomposition -- the attention front-end breaks the conservative gauge, and the SPLM back-end operates in a gauge-fixed space (Section 17b of the paper).
Why Not a Pure Transformer?
The Hybrid SPLM is a two-stage architecture that uses attention only in the front-end (4 blocks) and replaces the remaining computation with scalar-potential gradient dynamics (4 SPLM steps). Unlike a pure Transformer, the SPLM refinement stage has no KV-cache, no FFN towers, and is driven entirely by a single small scalar-potential MLP, \(V_\theta\) — 3-layer, 1024-hidden.
Key structural differences from Transformers:
Property
Transformer (GPT-2 small)
Hybrid SPLM (this model)
Architecture
12 self-attention + FFN blocks
4 attention blocks + 4 SPLM steps
Core computation
50.3M (MLP) + 28.3M (attention)
Attention front-end + 3.4M \(V_\theta\)
Runtime state per token
\(O(T)\) — full KV-cache
\(O(T)\) — KV-cache in attention stage
Total parameters
124M
~19.0M
Note: Because this model retains attention blocks in the front-end, its runtime memory is still \(O(T)\) in sequence length (due to the KV-cache in the attention stage). However, the SPLM tail contributes only \(O(d^2)\) per token, and at long context \(T \geq 4096\) the hybrid achieves -39% decode FLOPs vs matched pure attention. For fully \(O(1)\) inference, see the Multi-Xi SPLM, PARFLM, and Fock-PARFLM.
Runtime information capacity vs sequence length
Geometric Capabilities
Note: This model includes attention blocks in its front-end, breaking the conservative-by-construction guarantee. The full damped Riemannian geometry — layer-dependent Jacobi metric, damped geodesics with friction, energy dissipation anomaly detection, curvature proxy — is available only in the purely conservative variants: Multi-Xi SPLM, Multi-Xi PARFLM, and Fock-PARFLM v2.1. A Riemannian Geometry Diagnostic Battery (June 2026) confirmed the metric validity and characterised the damping-dominated dynamics across those models. See the companion note for the full damped framework.
However, the SPLM refinement layers in this hybrid architecture still use the same damped Euler integration with (V_\theta) as the purely conservative variants. The geometric structure (metric, geodesics, curvature) is well-defined within those layers — the attention front-end only prevents the global conservative guarantee from holding end-to-end.
How to Get Started
python
1# Clone the companion repository for full source code2# git clone https://github.com/dimitarpg13/semsimula-paper.git3# cd semsimula-paper/notebooks/conservative_arch45import torch
6import sys
7sys.path.insert(0,"hybrid")8sys.path.insert(0,"sarf_mass_variant")9sys.path.insert(0,".")1011from hybrid.model_hybrid import HybridSPLM, HSPLMConfig
1213config = HSPLMConfig(14 vocab_size=50257,# GPT-2 BPE15 d=256,16 n_attn=4,17 n_splm=4,18 n_head=4,19 v_hidden=1024,20 v_depth=3,21 max_len=1024,22 block_size=512,23 gamma_init=0.15,24)2526model = HybridSPLM(config)27print(f"Parameters: {sum(p.numel()for p in model.parameters()):,}")2829# Forward pass30x = torch.randint(0,50257,(1,64))31logits, loss = model(x, targets=x)
Available Checkpoint
A trained checkpoint (PPL 8.01, 16k steps) is included in this repository:
File
Description
checkpoint/model.pt
Full model state dict (73 MB)
training_log.jsonl
Per-step training metrics
loss_curve.png
Training/validation loss plot
v_theta_hist.png
V_theta histogram
landscape_stats.json
Landscape statistics (JSON)
To load the checkpoint:
python
1from huggingface_hub import hf_hub_download
2import torch
34# Download checkpoint5ckpt_path = hf_hub_download(6 repo_id="dimitarpg13/semsimula-hybrid-splm",7 filename="checkpoint/model.pt",8)910# Load into model (after creating model as above)11state = torch.load(ckpt_path, map_location="cpu")12model.load_state_dict(state["model_state_dict"])13model.eval()
Training Details
Training Data
TinyStories -- tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens; validation: ~140k tokens.
The Hybrid SPLM achieves the best PPL in the family (8.50), but uses attention and therefore does not satisfy the conservative-by-construction constraint that the pure SPLM / PARFLM / Fock variants maintain. It serves as the empirical bridge between attention and the fully conservative designs.
SPLM Family Overview
This model is part of the Semantic Simulation SPLM family:
