Code: mirrorethic/t3-reference (Apache-2.0)
Trace library + benchmarks: https://t3atlas.dev
Sibling checkpoint: mirrorethic/t3-124m-v36-pcloss — same architecture, trained with the inter-stage predictive-coding loss un-detached. Slightly worse PPL (28.53 vs 27.76), neutral on reasoning. Use the pair for the controlled inter-stage-PC ablation.

Quick start

pip install t3-reference

python

from huggingface_hub import hf_hub_download
from t3 import T3Model

ckpt = hf_hub_download("mirrorethic/t3-124m-v36", "pytorch_model.bin")
model = T3Model.from_checkpoint(ckpt)
model.eval()

import torch
input_ids = torch.randint(0, 50257, (1, 16))
with torch.no_grad():
    logits, *_ = model(input_ids)

To generate a schema-v1 ecology trace:

python

from t3.tracing import generate_trace
generate_trace(model, "The capital of France is",
               prompt_id="factual", n_tokens=32,
               out_path="trace.jsonl")

To re-run the published lm-eval-harness benchmarks:

python

from t3.benchmarks import run_benchmark_suite
results = run_benchmark_suite("path/to/pytorch_model.bin")

Architecture

T³ extends standard multi-head attention with a per-head ecology of six conjugate primitives (E, I, F, V, C, K) coupled through bivector composition in Cl(3,3) geometric algebra. Heads interact through a learned blockade-and-cosurvival graph and ponder adaptively per stage via output-entropy halt. Full technical specification: docs/ARCHITECTURE.md.

Field	Value
Parameters	124,500,000
Stages	3, with `layers_per_stage = [4, 3, 5]` (12 transformer blocks total)
`d_model`	768
`n_heads`	12
`d_ff`	3072
`vocab_size`	50257 (GPT-2 tokenizer)
`max_seq_len`	1024
Substrate	GPT-2 Small initialization
Training data	5B tokens (FineWeb-Edu 40%, DCLM 20%, StackEdu 10%, FineMath 10%, Cosmopedia 10%, Wikipedia 10%)
Cumulative training step	138,000 (135.5K substrate + 2,500 v3.6 increment)
Hamiltonian coupling ω	0.02
Trivectors	off (the trivectors-on variant is a planned v3.7 follow-up release)
Inter-stage predictive coding	on (`weight = 0.05`)
Scratchpad heads	on (`scratchpad_inject_entropy = (0.0, 0.0, 0.03)` — S2-only)
ACT	output-entropy halt + per-stage 4-step cap

Evaluation

All numbers are full lm-eval-harness 0.4.x runs (no subset). Reproduce with examples/run_benchmarks.py from the reference repo.

Task	Metric	Value	stderr
WikiText-103 (val)	perplexity	27.76	—
BoolQ	acc	0.6046	0.0086
ARC-Easy	acc	0.4331	0.0102
ARC-Challenge	acc	0.2176	0.0121
PIQA	acc	0.6050	0.0114
HellaSwag	acc	0.3040	0.0046
WinoGrande	acc	0.5043	0.0141
COPA	acc	0.6000	0.0492
RTE	acc	0.5235	0.0301

For comparison panels (parameter-efficiency vs vanilla GPT-2 same-data, compute frontier), see https://t3atlas.dev/benchmarks/.

vs vanilla GPT-2 (same 5B-token training data)

T³-124M-v36 vs gpt2/124M_vanilla_5b_same_data baseline. The interesting delta is on multi-step reasoning — T³ pondering allocates extra forward compute per token (S1 averages 2.7–3.7 ponder loops on these tasks).

Intended use

Research and interpretability. The checkpoint is designed to be inspected via the trace library, not deployed for production text generation. The model is small (124M), English-only, and not instruction-tuned.
Architectural comparison. A reference point for novel-attention work (Mamba, RWKV, xLSTM, etc.) that's matched to vanilla GPT-2 on training data and parameter count.
Ecology / dynamics analysis. The trace JSONL records per-head, per-stage ecology state across forward passes — useful for studying how Clifford-algebra-coupled state evolves during inference.

Limitations

124M parameters: too small to be a useful generative chat model.
English only.
No instruction tuning, no RLHF, no safety tuning.
Trained without grade-3 trivector terms (the static bivector Ω is the full Cl(3,3) rotation in this checkpoint). The state-dependent variant is a planned v3.7 Medium-scale follow-up.
The v36-pcloss sibling is the same architecture trained with the inter-stage predictive-coding loss un-detached. Slightly worse PPL (28.53), neutral on reasoning — the K-predictor learning a real cross-stage map (r=0.59) doesn't translate into downstream gains at this scale.

Capabilities probe

The checkpoint declares the following dynamics in its config and state dict (consumed by t3atlas viewer for trace-rendering):

json

{
  "has_coupling":       true,
  "has_trivectors":     false,
  "has_dyn_omega":      false,
  "has_inter_stage_pc": true,
  "has_scratchpad":     true,
  "n_primitives":       6,
  "null_cone_strength": 0.02,
  "hamiltonian_coupling": 0.02,
  "sigma_hidden":       16,
  "scratchpad_inject_entropy": [0.0, 0.0, 0.03]
}

Citation

bibtex

@misc{sutherland2026t3,
  author = {Sutherland, Garret},
  title  = {T³: A Clifford-Algebra-Augmented Transformer Architecture},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/mirrorethic/t3-124m-v36}
}

License

Apache-2.0. Both code (mirrorethic/t3-reference) and weights (this repository).

Contact

Garret Sutherland (MirrorEthic LLC) — gsutherland@mirrorethic.com.

Released 2026-05-03. The pytorch_model.bin here is a stripped inference-ready copy (498 MB) of the canonical best.pt from the v3.6 training campaign (run-3, step 2500, val PPL 27.76 on WikiText-103). The optimizer state and data-loader state were dropped; everything T3Model needs at inference is preserved (model_state, ecology_state, config, and provenance metadata).