You can deploy this model and start earning money today!

sida/
icalens-qwen3.5-9b-base-pile10k

Name: icalens-qwen3.5-9b-base-pile10k
Availability: InStock

icalens

interpretability

independent-component-analysis

activations

2606.11722

NeelNanda/pile-10k

Views

No views yet

ICA Lens for Qwen/Qwen3.5-9B-Base

This repository contains a fitted ICA Lens for analyzing internal activations of Qwen/Qwen3.5-9B-Base. It provides layer-wise ICA transformations for mapping residual-stream activations to independent-component scores and energy shares.

Artifact summary

Field	Value
Analyzed model	Qwen/Qwen3.5-9B-Base
Analyzed model revision	`68c46c4b3498877f3ef123c856ecfde50c39f404`
Model kind	`base`
Activation site	`resid_post`
Layer indexing	`transformer_blocks_zero_based`
Available layers	0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
Hidden size	4096
Input row normalization	`none`
ICALens package version	`0.3.6`
Fitting dataset	NeelNanda/pile-10k
Dataset revision	`127bfedcd5047750df5ccf3a12979a47bfa0bafa`
Dataset split	`train`
Fitting token scope	`all`
Candidate tokens	5256549
Fitting tokens	1000000

The analyzed model identity and revision are stored authoritatively in icalens.json; model-card metadata is not used when loading the lens.

Usage

Analyze text end to end:

python

from icalens import ICALens

lens = ICALens.from_pretrained("REPOSITORY_ID")
result = lens.analyze("She deposited the check at the bank.", layer=0)

print(result.tokens)
print(result.scores)  # signed standard ICA scores
print(result.energy)  # per-token squared-score fractions

Or transform activations captured separately:

scores = lens.transform(activations, layer=0)

Externally captured activations must use the model revision, activation site, layer indexing, and preprocessing recorded in icalens.json.

Score definition

Signed scores are the centered, whitened activations followed by the learned orthogonal ICA rotation. No post-ICA source scaling is applied to v0.2 fits. For a token, component energy share is score² / sum(all component scores²).

Fitting

Layer	Components	Fitting tokens	FastICA iterations
0	4096	1000000	50
1	4096	1000000	50
2	4096	1000000	50
3	4096	1000000	50
4	4096	1000000	50
5	4096	1000000	50
6	4096	1000000	50
7	4096	1000000	50
8	4096	1000000	50
9	4096	1000000	50
10	4096	1000000	50
11	4096	1000000	50
12	4096	1000000	50
13	4096	1000000	50
14	4096	1000000	50
15	4096	1000000	50
16	4096	1000000	50
17	4096	1000000	50
18	4096	1000000	50
19	4096	1000000	50
20	4096	1000000	50
21	4096	1000000	50
22	4096	1000000	50
23	4096	1000000	50
24	4096	1000000	50
25	4096	1000000	50
26	4096	1000000	50
27	4096	1000000	50
28	4096	1000000	50
29	4096	1000000	50
30	4096	1000000	50
31	4096	1000000	50

Fitting provenance

json

{
  "activation_dataset": {
    "dtype": "bfloat16",
    "format": "icalens.activations",
    "format_version": 1,
    "manifest_sha256": "f216c9b51600bbf69a2ecb2483f5db9e1f39ceb8d58178a8b09d5e701312926c"
  },
  "candidate_tokens": 5256549,
  "context_length": 1024,
  "dataset": {
    "repo_id": "NeelNanda/pile-10k",
    "revision": "127bfedcd5047750df5ccf3a12979a47bfa0bafa",
    "split": "train"
  },
  "document_framing": {
    "evidence_note": "Qwen documents are separated by <|endoftext|> inside packed pretraining sequences.",
    "evidence_url": "https://qwen.readthedocs.io/en/latest/getting_started/concepts.html#control-tokens",
    "included_in_fitting_samples": false,
    "policy_source": "project:model_framing.json",
    "registry_model_entry": "Qwen/Qwen3.5-9B-Base",
    "registry_schema_version": 1,
    "registry_sha256": "c6782bd5d550ed35b5e5b1e63c922907444bf4f40c10fb3f7f8f86858a4093d1",
    "requested": "auto",
    "strategy": "prepend-eos",
    "token": "<|endoftext|>",
    "token_id": 248044
  },
  "fitting_tokens": 1000000,
  "sampling_seed": 0,
  "text_field": "text",
  "token_scope": "all"
}

Limitations

Component IDs are specific to a layer and fitted artifact.
Standard ICA scores are signed and are not probabilities.

Paper

ICA Lens: Interpreting Language Models Without Training Another Dictionary

Citation

bibtex

@article{liu2026icalens,
  title={ICA Lens: Interpreting Language Models Without Training Another Dictionary},
  author={Liu, Sida and Han, Feijiang},
  journal={arXiv preprint arXiv:2606.11722},
  year={2026}
}

sida/icalens-qwen3.5-9b-base-pile10k