This is a set of transcoders trained on the
TinyStories-3M using the 2.2M sample of the
TinyStories dataset, which comes out to roughly 2.2B tokens using the full GPT-Neo tokenizer. We trained the transcoders on the MLP layers 2, 4, and 7 with a dictionary size 16-times that of the MLP layer. The transcoders are organized by hookpoint.
1from sae import Sae
2sae = Sae.load_from_hub("abmakkeh/transcoder-TinyStories-3M-16x", hookpoint="h.1.mlp")
