Phase-1 pre-pretraining only: Pythia-160M trained on English/C4 (500 steps). Intended as a starting point for C4 pretraining.
Part of an experiment reproducing and extending the pruning analysis from
"Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts
Linguistic Biases" (Hu et al., 2025,
arXiv:2502.19249).
The hypothesis under test: pre-pretraining on English itself yields attention-head
circuits as sparse/transferable as pre-pretraining on k-shuffle Dyck.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("gizemyc/pythia-160m-english-ppt-phase1")
4tokenizer = AutoTokenizer.from_pretrained("gizemyc/pythia-160m-english-ppt-phase1")
