Views
No views yet
talkie-lm/talkie-1930-13b-base
(13B, pre-1931 English) on real modern web text only: nvidia/ClimbMix
(docs ≤1024 GPT-2 tokens, detokenized → talkie BPE) mixed 1:1 with pre-1930
replay (PG-19), ~10B tokens total across 3 fresh-data epochs. LoRA r=128 on all
7 projections (never embed/lm_head) — adapter-off is bit-identical to the
base model. Method follows the SDF paper (arXiv:2510.17941) minus synthetic
docs: this run is the real-text baseline its synthetic V2 counterpart is
compared against.checkpoints/step_*.pt — log-spaced adapter snapshots (fp32; the RSA/geometry
ladder), checkpoints/final.pt — endpoint (~step 76300, ~10B tokens).
Load: apply_lora(model, r=128, alpha=256); load_adapter(model, ckpt["adapter"])
(see training_code/src/talkie_cpt/lora.py).eval/offline_battery*.log — P(fact) true/false margin battery per snapshot
(450 pairs from cds-jb/talkie-timetravel-synth factpairs), incl. obscurity
tiers; eval/skyline*.json — talkie-web-13b-base on the same battery.eval/geometry_ladder.json — per-layer cosine + linear CKA of adapted vs
base activations on held-out pre-1930 text, per snapshot.figures/ — training curves (log-log losses, P(fact) error incl. obscurity
tiers vs the talkie-web skyline), geometry-over-training.training_code/ — the complete talkie-cpt trainer + data pipeline used to
produce this run. Retrain: scripts/run_v1.sh (see its header).| group | stock | final | web skyline |
|---|---|---|---|
| cluster_berlin_wall | 0.378 | 0.711 | 0.889 |
| cluster_chernobyl | 0.356 | 0.644 | 0.778 |
| cluster_moon_landing | 0.422 | 0.800 | 0.889 |
| cluster_penicillin | 0.467 | 0.733 | 0.844 |
| post1930 | 0.494 | 0.806 | 0.983 |
| pre1930 | 0.922 | 0.956 | 0.978 |