You can deploy this model and start earning money today!

kevinjin918/
clt-medgemma-cxr

Name: clt-medgemma-cxr
Availability: InStock

pytorch

cross_layer_transcoder

Views

No views yet

Cross-Layer Transcoder for MedGemma (chest X-ray)

The first image-token-aware windowed cross-layer transcoder (CLT) trained on MedGemma activations over paired chest-X-ray and report data. Trained as part of TraceCXR, a mechanistic-interpretability investigation into report-prior hallucination — a vision-language model asserting findings its image does not support.

What it is

A sparse dictionary that reconstructs a layer's activations from a sparse feature basis, reading across a window of layers rather than a single one. Unlike a per-layer SAE, each feature can write into several downstream layers, which is what makes cross-layer circuit tracing possible.


base model	MedGemma (`d_model` = 2560)
layers covered	12
features	8,192
decoder span	3 layers
training steps	20,000
precision	bf16 training, fp32 weights

Weights (params):

tensor	shape
`W_enc`	(12, 2560, 8192)
`b_enc`	(12, 8192)
`W_dec`	(12, 3, 8192, 2560)
`b_dec`	(12, 2560)

Reconstruction quality

Fraction of variance unexplained (FVU) on held-out activations — lower is better:

stream	FVU
text	0.083
image	0.163
combined	0.083

L0 (mean active features per token) ≈ 30. Reconstruction MSE ≈ 6e-4.

Image activations reconstruct less well than text, which is itself informative: the image stream carries more variance that a sparse basis of this width does not capture.

Loading

python

import torch
ckpt = torch.load("clt_medgemma_cxr.pt", map_location="cpu")
W_enc, b_enc = ckpt["params"]["W_enc"], ckpt["params"]["b_enc"]   # (12, 2560, 8192), (12, 8192)
W_dec, b_dec = ckpt["params"]["W_dec"], ckpt["params"]["b_dec"]   # (12, 3, 8192, 2560), (12, 2560)

# per layer L: features = relu(acts_L @ W_enc[L] + b_enc[L])
# the decoder then writes into layers L .. L+2 via W_dec[L]

Optimizer state has been stripped (the original checkpoint was 12GB; this is 4GB of weights only).

Honest limitations

Reconstruction quality is not interpretability. A low FVU says the dictionary spans the activations, not that its features are monosemantic or causally meaningful. No feature-level interpretability audit is published with this checkpoint.
The TraceCXR finding this was built to test — that report-prior hallucination is distributed rather than a clampable circuit — is a negative result. This transcoder is released as a reusable artifact, not as evidence that the intervention worked.
Trained on one base model and one domain. Transfer to other VLMs or modalities is untested.

Citation

Part of the TraceCXR project (Kevin Jin, Duke University, 2026).

kevinjin918/clt-medgemma-cxr