This repository publishes the requested ordinary checkpoint release for step_14500, the current best behavior-only candidate inside the replayed medium GPT2PreLN 13.5k decay family.

This is still not the definitive public-facing family base release.

Released checkpoint summary:

released checkpoint: step_14500.pt
replay continuation run:
- 20260629_resume-gpt2medium-gpt2preln-k20-wsddecayonly-rerunmissing-lr3p5294e5-anchor20k-final2e5-webwiki-step14200-to14850
original decay-only parent run:
- 20260628_resume-gpt2medium-gpt2preln-k20-wsddecayonly-lr2e-4-anchor20k-final2e5-webwiki-step13500
parent non-decayed anchor:
- stable-recipe-gpt2medium-gpt2preln-k20-wsd-lr2e-4-anchor20k-final2e5-webwiki
original continuation source checkpoint:
- step_13500.pt
replay source checkpoint:
- step_14200.pt
languages: English + Italian
context window: 2500 tokens
architecture: GPT-2-style decoder with pre-layernorm blocks
architecture config: architecture: gpt2, block_type: gpt2_prelayernorm
training-time parameter count: 337,671,424
published Transformers-export parameter count: 337,639,424
hardware class: single consumer GPU (RTX 4060 Ti 16GB) for training, GPU benchmarked selection

This is a base pretraining checkpoint, not an instruction-tuned chat model.

Why This Checkpoint Exists

step_14500 is not trying to win the scalar leaderboard. It exists because it is the cleanest behavior-oriented compromise in the replay group when the selection pressure is aimed at lower looping and higher local diversity.

So this release is for live testing and qualitative inspection when the goal is "show me the best behavior-only candidate", even if that means giving back some scalar loss against 14250 and 14700.

Position Inside The Decay Family

The important scoreboard is:

replay step_14250: val_loss_mixed = 4.4419
replay step_14700: val_loss_mixed = 4.4436
replay step_14500: val_loss_mixed = 4.4926
replay step_14850: val_loss_mixed = 4.4521
source non-decayed step_13500: val_loss_mixed = 4.4652

Behavior snapshot:

step_14500: loop_rate = 0.350, distinct_2 = 0.5822, repeated_4gram_rate = 0.650, lang_en = 1.000, lang_it = 0.800
step_14700: loop_rate = 0.375, distinct_2 = 0.5643, repeated_4gram_rate = 0.750, lang_en = 1.000, lang_it = 0.850
step_14250: loop_rate = 0.425, distinct_2 = 0.5617, repeated_4gram_rate = 0.775, lang_en = 0.950, lang_it = 0.825

So the honest read is:

step_14500 is clearly behind the scalar winners
but it is the most appealing behavior-only candidate in the replay set
it has the best distinct_2 of the replay family, a low loop_rate, and the lowest repeated_4gram_rate among the short-listed replay checkpoints
this is the checkpoint to use when the goal is "best behavior candidate to check live", not "best measured scalar release"

Main Metrics For `step_14500`

val_loss_mixed = 4.4926
val_loss_en = 4.4393
val_loss_it = 3.6251
ppl_mixed = 89.3505

Behavior snapshot:

loop_rate = 0.350
distinct_2 = 0.5822
repeated_4gram_rate = 0.650
language_consistency_en = 1.000
language_consistency_it = 0.800
cloze_en_contains = 0.14
cloze_it_contains = 0.20

Short read:

this is the cleanest replay-tail checkpoint when the priority is behavior
it pays for that with weaker scalar loss than 14250, 14700, and 14850
this is the checkpoint to inspect when the question is qualitative balance, not leaderboard placement

Training Data

This model was trained on the bilingual EN/IT web + wiki dataset:

dataset id on disk:
- 202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M
context window during training: 2500 tokens
packing length: 2500
mixing strategy: source_balanced
validation ratio: 0.05

Main source groups:

English FineWeb-HQ (epfml/FineWeb-HQ)
Italian FineWeb2-HQ (epfml/FineWeb2-HQ)
English Wiki40B (google/wiki40b)
Italian Wiki40B (google/wiki40b)

How Many Tokens This Checkpoint Saw

Training math:

sequence length: 2500
batch size: 2
grad accumulation: 48
tokens per optimizer step: 239,904

So step_14500 saw approximately:

3.4786080B tokens total
K = 10.3018 tokens per parameter relative to the native training-time parameter count

Included Files

This release bundle includes:

step_14500.pt
step_14500.safetensors
model.safetensors
config.json
tokenizer files
training_config.yaml
run telemetry:
- best_validation.json
- metrics.jsonl
- eval_metrics.jsonl
- probe_generations.jsonl
benchmark bundle:
- summary.json
- comparison.json
- comparison.csv
- metrics.json
- metrics.csv
- source_losses.json
- report.md
- generations.jsonl
- generations_comparison.md
- cloze_results.jsonl

Quick Start

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo_id = "nazdef/gpt2medium-en-it-nanochat-gpt2preln-decay13500-step14500"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "La capitale d'Italia è"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)

outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    do_sample=True,
    max_new_tokens=64,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

This release is published with CC-BY-SA-4.0 as the practical downstream posture for the mixed training corpus used here.

The training mix includes:

FineWeb-HQ / FineWeb2-HQ web data
Wiki40B English and Italian slices

Downstream users are responsible for checking whether their use, redistribution, or derivative packaging remains compatible with the obligations of the upstream datasets and their terms.

nazdef/gpt2medium-en-it-nanochat-gpt2preln-decay13500-step14500

GPT2Medium EN/IT NanoChat GPT2PreLN decay-only step_14500