Base autoregressive → MLM diffusion models derived from the EleutherAI Pythia family.
These are base models only (no supervised fine-tuning / instruction tuning).

Model description

Model type: Decoder-only Transformer converted to Masked Language Modeling (MLM) diffusion architecture
Base architecture: Pythia (GPT-NeoX)

Variants

Size	HF Base	Checkpoint	Total Params	Non-embedding Params	Tokens seen (pretrain)	Tokens / Param
70M	`EleutherAI/pythia-70m`	step 512	70.4M	18.9M	~1.07B	~15 + continued
160M	`EleutherAI/pythia-160m`	step 1000	~162M	~125M	~2.1B	~13 + continued
410M	`EleutherAI/pythia-410m`	step 3000	405.3M	~350M	~6.29B	~15.5 + continued

Original Pythia pretraining & checkpoint selection (EleutherAI):
All models were initialized from Pythia checkpoints selected according to Chinchilla scaling laws (Hoffmann et al., 2022) targeting ~15 tokens per parameter (slightly undertrained regime) for consistency across the model family.

Dataset: The Pile
Total tokens in Pythia run: 299.9B
Checkpoint interval: every 2,097,152 tokens (~2M batch size)

Selected checkpoints:

70M: step 512 (~1.07B tokens, ~15 tok/param)
160M: step 1000 (~2.1B tokens, ~13 tok/param)
410M: step 3000 (~6.29B tokens, ~15.5 tok/param)

Intended uses & limitations

Research on autoregressive-to-diffusion conversion
Continued pre-training and domain adaptation experiments
Analysis of scaling behavior in hybrid MLM diffusion models
Not intended for direct deployment or chat/instruction use (base models only)

Training and evaluation data

Continued pre-training: All models were further trained on the same subset of ~3B tokens sampled from FineWeb-Edu using (masked) causal language modeling objective.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 48
eval_batch_size: 48
seed: 46
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.85,0.94) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 625
training_steps: 6250

How to Get Started with the Model

Use the following code to get started with the model:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "avgJo3/a2d-gpt-neox-160M"  # or 70m / 160m variant
revision = "main"  # or specific checkpoint tag if uploaded separately

model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    revision=revision, 
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    model_name, 
    revision=revision, 
    trust_remote_code=True
)

Framework versions

Transformers 5.0.0
Pytorch 2.10.0+cu128
Datasets 4.8.5
Tokenizers 0.22.2

Citation

bibtex

@article{biderman2023pythia,
  title={Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling},
  author={Biderman, Stella and others},
  journal={ICML},
  year={2023}
}

avgJo3/a2d-gpt-neox-400M

a2d-gpt-neox-400M