Repository: Saraquel/AufhebenAdapter
The Aufheben Adaptor is a formal implementation of a dialectical attention mechanism. It translates Hegelian sublation (Aufheben) into a measurable vector space. By running two mathematically incongruent attention heads over the same value space, the adaptor forces the base model to simultaneously commit to a structural context (affirmation) and explicitly flag what that context excludes (negation).
This allows the model to calculate the exact intensity of its own internal contradictions and use that tension to dynamically steer its latent states away from structural failure modes—without altering the frozen base model's weights.
Core Architecture: The Dialectic in Latent Space
The key architectural property of the Aufheben Adaptor is the mathematical incongruence between its Positive and Negative attention heads. They evaluate the exact same value space but apply fundamentally different activation functions.
- Positive Attention: Softmax
The positive head uses a standard softmax function. Softmax forces the attention scores to compete and normalizes them to sum exactly to 1.
- This encodes what is structurally present, coherent, and selected.
- Negative Attention: Sigmoid
The negative head evaluates the identical value space but uses a sigmoid function. Sigmoid evaluates each position independently; the scores are non-competitive and do not sum to 1.
- This encodes what is structurally excluded or suppressed. It measures the diffuse, non-zero suppression signal representing relationships the model is not committing to.
Dasien: The Logic Vector ($L$)
A gate mechanism mediates (but does not resolve) the tension between the positive and negative streams. The sublation is calculated as the residual tension, producing the Logic Vector:
$$L = P \odot G - N \odot (1 - G)$$
When affirmation and negation strongly oppose each other, $||L||$ becomes large. This vector is the literal, mathematical intensity of the dilemma, which is then pooled to calculate a Danger Score and injected back into the hidden states to steer the generation.
Critical Note on Precision: FP32 is Required
Do not run the adaptor in FP16 (16-bit floating-point). The adaptor requires FP32.
The entire mechanism relies on calculating the residual tension between the $P$ and $N$ vectors. Because these vectors are often very close in magnitude, calculating $L$ requires high numerical granularity.
If you lower the precision to FP16, the model forces these continuous values into wider numerical "buckets."
- Catastrophic Cancellation: Granular differences round to the same number, artificially dropping the tension to zero.
- Artificial Spikes: If numbers land on the edge of quantization buckets, FP16 can artificially inflate the difference. Because this inflated tension is subsequently passed through a sigmoid function (which is highly sensitive to inputs around 0) and then squared for the steering multiplier ($\alpha$), rounding errors will result in violent, hallucination-inducing steering injections.
Note on bfloat16: Theoretical alternatives like bfloat16 offer a wider dynamic range than standard FP16, but they still truncate precision. This has not been formally tested with the adaptor and is generally not recommended unless you are prepared to manage severe measurement distortion.
Structural Tension at Initialization
Because the architectural tension relies purely on the incongruence between softmax and sigmoid functions over a shared value space, the axis of contradiction exists before any training occurs.
At random initialization, softmax immediately produces a normalized, competitive distribution, while sigmoid naturally clusters near 0.5, producing a diffuse suppression signal. Therefore, the adaptor will detect and encode structural tension right out of the box. Training the adaptor merely sharpens this existing axis; it does not create it.
Repository Structure
AufhebenAdapter.pth (190 MB): The specific, trained weights for the dialectical adaptor (gate, positive/negative projection matrices, and injection MLPs).Neg_Pos_Heads.pt (5.77 GB): Contains the heavier tensors utilized for the positive and negative attention state dictionaries..gitattributes: Git LFS tracking configuration.
Citation
If you use this adapter, the architecture, or the findings from the "Impossible Crisis" experiments in your work, please cite it as follows:
1@software{cuevas_uriostique_2026_hegelian,
2 author = {Alexis Cuevas Uriostique},
3 title = {Dialectical Synthesis: Alignment via Geometric Differentiation},
4 month = {April},
5 year = {2026},
6 url = {[https://huggingface.co/your-username/hegelian-phi2](https://huggingface.co/your-username/hegelian-phi2)}
7}
8
9
10
11## Usage
12
13*This adaptor is designed to operate on top of a frozen base model Phi - 2, although you may train the architecture on any model to get the adapter trained on the latent space dynamics. Ensure your base model hidden states are accessible for additive latent steering prior to the language modeling head.*
14
15To maintain the **Asymptotic Steering** stability and avoid numerical overflow, the adapter must be loaded in **FP32**.
16
17```python
18import torch
19from transformers import AutoModelForCausalLM
20
21# 1. Load the frozen base model
22model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype=torch.float16)
23
24# 2. Initialize and load the Hegelian Adapter in FP32
25adapter = AufhebenUnified(hidden_size=2560)
26adapter.load_state_dict(torch.load("hegelian_adapter.pt"))
27adapter.float() # <--- CRITICAL: Manually cast to FP32 for high-norm steering
28adapter.to("cuda")
