This is a set of transcoders trained on a
GPT2 implementation of TinyStories-1M using the 2.2M sample of the
TinyStories dataset, which comes out to roughly 2.2B tokens using the full GPT2 tokenizer. We trained transcoders of different sizes on the MLP layers 2,4, and 7, with a dictionary size twice that of the MLP layer. The transcoders are organized by hookpoint.
1from sae import Sae
2sae = Sae.load_from_hub("abmakkeh/transcoder-TinyStories-1M-2x", hookpoint="h.1.mlp")
