Parameter	Value
Base architecture	EleutherAI/pythia-160m (reinitialized)
Regimen	`control_shuffle_dyck_steps500`
Seed	208
Stage 1 dataset	Shuffled Shuffle-Dyck (unstructured control)
Stage 1 steps	500
Stage 2 dataset	OpenWebText
Stage 2 steps	10000
Optimizer	AdamW (lr=1e-3, wd=0.0)
Effective batch size	64
Sequence length	2048

Control Design

Stage 1: Shuffle-Dyck tokens with shuffled order. Same token distribution as shuffle_dyck but no sequential structure. This is the unstructured control.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("sashaboguraev/pythia-160m-ppt-control_shuffle_dyck_steps500-seed208")
tokenizer = AutoTokenizer.from_pretrained("sashaboguraev/pythia-160m-ppt-control_shuffle_dyck_steps500-seed208")

Citation

If you use this model, please cite the original pre-pretraining papers:

Papadimitriou & Jurafsky (2020) — tilt-transfer
Hahn & Rofin (2024) — pre-pretraining with formal languages (michahu)
Lee et al. (2024) — NCA pre-pretraining (danihyunlee)

sashaboguraev/pythia-160m-ppt-control_shuffle_dyck_steps500-seed208

Pythia-160M Pre-Pretraining: control_shuffle_dyck_steps500 (seed 208)

Training Details

Control Design

Usage

Citation

sashaboguraev/
pythia-160m-ppt-control_shuffle_dyck_steps500-seed208

Pythia-160M Pre-Pretraining: `control_shuffle_dyck_steps500` (seed 208)