Component	Description	Params
GeometricProcessor	4-layer causal transformer with KV caching producing additive `geo_logits`	~148M
LatentPlanner	VAE with LaDiR-style diffusion ELBO for planning latent z₀	~14M
EBM Critic	Energy-based model scoring geometric sequence quality	~0.5M
Alpha Gate	Learned sigmoid gate (α=0.537) blending sidecar corrections	1

Forward pass: final_logits = base_logits + α · geo_logits

Training Details

Backbone: google/gemma-4-E2B-it (frozen, ~2.6B params)
Sidecar: 162,291,818 trainable parameters
Training data: HuggingFaceH4/ultrachat_200k (streaming)
Hardware: NVIDIA H100 80GB HBM3
Checkpoint: Epoch 2, Step 6250
SVD initialization: GeometricProcessor input projection initialized with top-512 PCA components of backbone embeddings

Usage

python

# Install
!pip install transformers accelerate huggingface_hub torch

# Clone the repo
!git clone https://github.com/Bender1011001/dual-system-architecture.git

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from dual_system_v2 import DualSystemV2, SidecarConfig
from huggingface_hub import hf_hub_download

# Download sidecar
sidecar_path = hf_hub_download(
    repo_id="Bender1011001/gemma4-dualsystem-sidecar",
    filename="sidecar_epoch2.pt"
)

# Load checkpoint config (guarantees weight compatibility)
ckpt = torch.load(sidecar_path, map_location="cuda", weights_only=False)
config = SidecarConfig(**ckpt["config"])

# Load backbone
backbone = AutoModelForCausalLM.from_pretrained(
    ckpt["backbone"], torch_dtype=torch.bfloat16, device_map="cuda"
)
for p in backbone.parameters():
    p.requires_grad = False

# Build and load sidecar
model = DualSystemV2(backbone=backbone, config=config).cuda().eval()
model.geo_processor.load_state_dict(ckpt["geo_state"])
model.ebm_critic.load_state_dict(ckpt["ebm_state"])
model.latent_planner.load_state_dict(ckpt["planner_state"])

# Generate
tokenizer = AutoTokenizer.from_pretrained(ckpt["backbone"])
result = model(input_ids=tokenizer("Hello", return_tensors="pt").input_ids.cuda())

Colab Demo

Open In Colab

Citation

bibtex

@misc{dual-system-2026,
  title={Dual-System Architecture: Geometric Sidecar Modules for Language Model Enhancement},
  author={Bender1011001},
  year={2026},
  url={https://github.com/Bender1011001/dual-system-architecture}
}