This is a set of transcoders trained on a
GPT2 implementation of TinyStories-1M using the 2.2M sample of the
TinyStories dataset, which comes out to roughly 2.2B tokens using the full GPT2 tokenizer. We trained the transcoders on the MLP layers 2, 4, and 7 with a dictionary size 16-times that of the MLP layer. The transcoders are organized by hookpoint.
1from sae import Sae
2sae = Sae.load_from_hub("abmakkeh/transcoder-TinyStories-1M-16x", hookpoint="h.1.mlp")
