You can deploy this model and start earning money today!

gizemyc/
pythia-160m-c4-english-ppt

Name: pythia-160m-c4-english-ppt
Availability: InStock

EleutherAI/pythia-160m

finetune

apache-2.0

Views

No views yet

pythia-160m-c4-english-ppt

Pythia-160M pre-pretrained on English (C4), then pretrained on C4.

Part of an experiment reproducing and extending the pruning analysis from "Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases" (Hu et al., 2025, arXiv:2502.19249).

The hypothesis under test: pre-pretraining on English itself yields attention-head circuits as sparse/transferable as pre-pretraining on k-shuffle Dyck.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gizemyc/pythia-160m-c4-english-ppt")
tokenizer = AutoTokenizer.from_pretrained("gizemyc/pythia-160m-c4-english-ppt")

Training

Base architecture: EleutherAI/pythia-160m (12 layers x 12 heads)
Pretraining data: C4 (English)
Pre-pretraining (where applicable): k-shuffle Dyck or English/C4