You can deploy this model and start earning money today!

markhenry/
msae-3L-on-cayley-v1

Name: msae-3L-on-cayley-v1
Availability: InStock

Views

No views yet

msae-3L-on-cayley-v1

3-level Matryoshka BatchTopK SAE (dict=21504, k=48, groups=[1024,4096,16384]) trained on the output of a CayleySAE (mlp_in_post_sae, layer 6) of vanilla-loss-parity-v2. Part of a matched triplet comparing cayley-output / cayley-input / vanilla activations.

Training details

trainer: MatryoshkaBatchTopKTrainer
dict_size: 21504
k: 48
group_sizes: [1024, 4096, 16384]
activation_dim: 1024
layer: 6
location: mlp_in_post_sae
base_model: out/cayley-32k-2L-mlp_in-v1
steps: 97656
lr: 5e-05

Usage

python

from dictionary_learning.trainers.matryoshka_batch_top_k import MatryoshkaBatchTopKSAE
sae = MatryoshkaBatchTopKSAE.from_pretrained('ae.pt', device='cuda')
# encode: sae(x) returns (recon, f, loss_dict)
# decode: sae.decode(f)

Part of a research project comparing Matryoshka SAEs and CayleySAEs on hierarchy properties. See evals/msae/ in sparse-nanogpt-private.

markhenry/msae-3L-on-cayley-v1

msae-3L-on-cayley-v1

Training details

Usage

markhenry/
msae-3L-on-cayley-v1