Real-model validation is pending (Exp-111). Exp-110 results use a mock LLM
with deterministic error injection. The constraint checker works correctly
(0.006 ms/check on CPU); the guidance logic is unvalidated on live models.
guided-decoding-adapter
Energy-guided decoding adapter for any HuggingFace causal LM.
Attaches Carnot's constraint energy pipeline to the token generation loop.
Each token step runs a constraint violation check on the text generated so far;
violating tokens are penalised by subtracting alpha × violation_count from
all logits before sampling.
How It Works
prompt → encode → [forward pass → check constraints → penalise logits → sample] × N → text
The constraint checker (AutoExtractor) detects violations across four domains:
Energy is a plain violation count (not a calibrated probability). The penalty
is applied uniformly across the vocabulary — token ranking is preserved while
overall entropy increases, discouraging the model from continuing down a
constraint-violating path.
Latency Profile
From Exp-102 (CPU, JAX_PLATFORMS=cpu, 1000-iteration benchmark):
Measurement
Value
Constraint check p50
0.006 ms
Constraint check p99
0.034 ms
Extraction p50
0.276 ms
Per-token budget fraction
0.04% of 20 ms/token
Verdict
Fits in real-time generation budget
Usage
python
1from carnot.inference.guided_decoding import GuidedDecoder
2from transformers import AutoModelForCausalLM, AutoTokenizer
34# Load model (any HF causal LM)5model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B")6tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")7model.eval()89# Load adapter from local directory or HuggingFace Hub10decoder = GuidedDecoder.from_pretrained("Carnot-EBM/guided-decoding-adapter")1112# Generate with constraint guidance13result = decoder.generate(model, tokenizer,"What is 47 + 28?")14print(result.text)15print(f"Energy checks: {result.energy_checks}, final energy: {result.final_energy}")
Override defaults
python
1decoder = GuidedDecoder.from_pretrained(2"Carnot-EBM/guided-decoding-adapter",3 alpha=1.0,# stronger guidance4 check_every_k=5,# check every 5 tokens (faster, less precise)5 energy_threshold=0.5# only penalise when violations > 0.56)
Any HuggingFace AutoModelForCausalLM with .logits output should work.
The adapter does not modify model weights.
Benchmark Results (Exp-138 & Exp-140)
Note — Simulated Inference: All benchmark numbers below were produced
with a simulated (mock) LLM, not a real transformer model. The constraint
checker and logit-penalty logic are real; the generation loop uses a
deterministic stand-in. Live-model E2E validation is pending (Exp-111).
Latency — KAN Projection Mode (Exp-140, batch=1, CPU)
Operation
p50
p99
Logit projection (energy gradient)
0.077 ms
0.271 ms
Total per-token (grad + projection)
0.405 ms
0.924 ms
Exp-140 pass criterion: total p50 < 5 ms — PASSED
(actual 0.4054 ms vs 5.0 ms threshold).
Installation
pip install carnot
Requires Python 3.11+. See pypi.org/project/carnot
for the full package including the verify-repair pipeline.
Limitations
Simulated inference benchmark: Exp-138 and Exp-140 used a mock LLM.
Numbers show constraint-checker and logit-penalty overhead, not end-to-end
accuracy on real models. Treat accuracy deltas as directional, not final.
No KV-cache: Full forward pass every token. Keep max_tokens < 256.
Uniform penalty: Adjusts entropy across the whole vocabulary; does not
steer towards specific correct tokens.
Energy is a violation count: Not a calibrated probability. High alpha
many violations → very flat distribution (model may repeat or stall).
For production LLM output verification, install: pip install carnot
This model is a Phase 1 research artifact (activation-based confidence detection).
These 16 per-token activation EBMs detect confidence, not correctness.
For the full verify-repair pipeline, see: https://github.com/Carnot-EBM/carnot-ebm
