The source layers follow the default fitting configuration of the reference J-Lens implementation. For a 32-layer model, the final transformer layer (layer 31) is used as the target layer, and all preceding layers (0–30) are used as source layers.

Files

`lens.pt`

The final fitted Jacobian Lens.

This is the main file intended for downstream J-Lens inference and interpretability experiments.

`lens.pt.ckpt`

A resumable checkpoint from the J-Lens fitting process.

It stores the accumulated fitting state and can be used to resume or extend fitting. This file is not required when only using the final fitted lens for analysis.

`lens.pt.metadata.json`

Metadata describing the model, fitting corpus, source layers, and fitting configuration used to generate the lens.

Usage

With the reference jlens implementation installed, the fitted lens can be loaded directly from this Hugging Face repository:

python

from jlens import JacobianLens

lens = JacobianLens.from_pretrained(
    "da3eun/tofu-llama-3.1-8b-jlens-wikitext"
)

print(lens)

Only lens.pt is downloaded when loading the lens with JacobianLens.from_pretrained() using its default filename.

Method

The Jacobian Lens reads out an intermediate residual-stream representation by linearly transporting it into the final-layer representation space using an average input-output Jacobian and then decoding it with the model's own unembedding.

For each fitted source layer (l), the lens contains an averaged Jacobian matrix (J_l) that maps the residual representation at layer (l) into the final-layer basis.

The methodology and implementation follow the Jacobian Lens introduced in:

Verbalizable Representations Form a Global Workspace in Language Models

Reference implementation:

anthropics/jacobian-lens

Intended Use

This artifact is intended for research on language-model interpretability, particularly for analyzing intermediate hidden representations of:

open-unlearning/tofu_Llama-3.1-8B-Instruct_full

The lens was fitted for experiments involving intermediate-representation decoding and comparison with other lens-based analysis methods.

This repository contains J-Lens artifacts only and does not redistribute the underlying Llama or TOFU model weights.

Notes

The lens was fitted on WikiText rather than the TOFU evaluation set.
The lens is model-specific and is intended for use with open-unlearning/tofu_Llama-3.1-8B-Instruct_full.
lens.pt is sufficient for standard J-Lens analysis.
lens.pt.ckpt is included for reproducibility and for resuming the fitting process.
Layer 31 is the target layer and therefore is not included among the fitted source layers.
The fitting metadata is also provided separately in lens.pt.metadata.json.

Attribution and Licenses

The Jacobian Lens reference implementation is released under the Apache License 2.0.

The fitting corpus, Salesforce/wikitext, is distributed under its applicable WikiText licensing terms.

The base model used to fit this artifact is based on Meta Llama 3.1 and is subject to the applicable licensing and usage conditions of the corresponding model and Llama 3.1.

Built with Llama.

Users of this artifact are responsible for reviewing and complying with the licenses and usage conditions of the underlying model, dataset, and reference implementation.