We introduce the Hayula Neural Transformer (HNT), a novel architecture that bakes Global Workspace theory (Baars, 1988) into transformer design through Multi-Scale Residual Streams. Inspired by the discovery that J-lens vectors form a global workspace in standard transformers—with broadcast heads specialized for relaying workspace content and MLP layers preferentially amplifying workspace-aligned directions (Gurnee et al., 2026)—HNT explicitly separates computation into three residual streams: a main stream for standard autoregressive computation, a local stream for syntax and surface-level features concentrated in early layers, and a global stream for workspace-level conceptual content concentrated in middle layers. A Cross-Stream Gating (CSG) mechanism, parameterized as a low-rank bilinear form inspired by QK-circuit analysis (Anthropic, 2025a), controls information flow between streams while minimizing feature interference. We further introduce Broadcast Heads: attention heads whose OV circuits are optimized for workspace content relay using gain-based selection metrics derived from MLP amplification analysis. Applied to security-domain training on a curated corpus of threat intelligence, vulnerability descriptions, and secure code repositories, HNT achieves 18% greater concept depth (measured by J-lens projection magnitude) than equivalent-parameter standard transformers, with a 22% reduction in feature interference as measured by the interference weight metric (Anthropic, 2025b). Our results demonstrate that architectural specialization informed by mechanistic interpretability yields disproportionate gains in domain-specific reasoning tasks.
The full paper is available as
hnt-architecture.md in this repository.
1@techreport{hayula2026hnt,
2 title={Hayula Neural Transformer: Multi-Scale Residual Stream Architecture with Global Workspace Gating for Security Language Models},
3 author={Hayula AI Lab},
4 year={2026},
5 month={July},
6 note={Preprint}
7}
