You can deploy this model and start earning money today!

tim-lawson/
mlsae-gemma-2-2b-x64-k32

Name: mlsae-gemma-2-2b-x64-k32
Availability: InStock

pytorch_model_hub_mixin

mit

Views

No views yet

Model Card for tim-lawson/mlsae-gemma-2-2b-x64-k32

A Multi-Layer Sparse Autoencoder (MLSAE) trained on the residual stream activation vectors from google/gemma-2-2b with an expansion factor of R = 64 and sparsity k = 32, over 1 billion tokens from monology/pile-uncopyrighted.

This model is a PyTorch TopKSAE module, which does not include the underlying transformer.

Model Sources

Repository: https://github.com/tim-lawson/mlsae
Paper: https://arxiv.org/abs/2409.04185
Weights & Biases: https://wandb.ai/timlawson-/mlsae

Citation

BibTeX:

bibtex

@misc{lawson_residual_2024,
  title         = {Residual {{Stream Analysis}} with {{Multi-Layer SAEs}}},
  author        = {Lawson, Tim and Farnik, Lucy and Houghton, Conor and Aitchison, Laurence},
  year          = {2024},
  month         = oct,
  number        = {arXiv:2409.04185},
  eprint        = {2409.04185},
  primaryclass  = {cs},
  publisher     = {arXiv},
  doi           = {10.48550/arXiv.2409.04185},
  urldate       = {2024-10-08},
  archiveprefix = {arXiv}
}