You can deploy this model and start earning money today!

ncylich/
temporal-moe-extras

safetensors

2505.20225

Views

No views yet

temporal-moe-extras

Supporting artifacts for the Temporal-MoE experiments, 248 files, 39.78 GiB. These are things that are expensive to regenerate but are not themselves training checkpoints.

Layout

olmoe_adapt/
  ckpt_bake_{CE,G,H,Er8,Er64,D}.pt   full optimizer and training state behind the router bakes
  finetune_ids.pt                    tokenized finetune corpus token ids
  bpb_slice_ids.pt                   bits-per-byte evaluation slice
  cal2_gprime.pt                     calibration tensors
  oseries/, oseries_ce/              paired evaluation slices, val.f16, idx.u8, paired masks
  logs/                              bake, sweep, calibration and evaluation logs
merged_ce_model/                     OLMoE backbone merged with the bake-CE router, loadable with
                                     transformers, includes config and tokenizer
run_captures/<run_name>/
  router_log.pt                      per-run router traces, back the mechanistic interpretability
                                     and residency result tables
  delex_capture.pt                   delexicalization captures
ablations/                           101 result tables as CSV, plus FINDINGS.md and README.md
figures/                             43 figures
tokenizer_tok16k/                    the 16k tokenizer, also mirrored in the corpus repository

Notes

merged_ce_model/ is a derived artifact. It is included rather than regenerated because the merge is not guaranteed to be bit-reproducible.

ablations/ holds the result tables the paper draws on. FINDINGS.md and README.md in that directory are the written interpretation of those tables.

The base OLMoE model is not mirrored here. It is public at allenai/OLMoE-1B-7B-0924.

Training configuration

Every run directory contains run.meta and train.log. run.meta records the full knob set for that run in two lines, for example:

[run] temporal_fine_g3_1e19 N=185766400 iters=17112 warmup=856 min_lr=0.0003 eval@1711
[run] shape=s19opt H=800 L=14 ffn=4272 moe_ffn=184 grain=3 num_experts=192 shared_int=1100
      dense=0 temporal=1 shared_mult=2 topk=18 heads=50 gb=256 mb=8 lr=3e-3 flops=1e19

train.log is the full Megatron stdout, including the complete argument namespace dumped at startup. Read these before drawing any conclusion from a checkpoint.

MANIFEST.csv and the `cited` column

A manifest covering every file in all four repositories lives in the code repository. It has seven columns: local_path, hf_repo, hf_path, bytes, sha256, run_name, cited.

The cited column marks whether a run is referenced by a results table in results/ablations/*.csv or by the paper:

cited, the run backs a published number. There are 58 of these.
uncited, the run is infrastructure validation, a smoke test, a throughput probe, or an aborted run. It is kept for completeness and reproducibility, not because a table depends on it. There are 13 of these.
empty, the file is not scoped to a single run, for example a batch log or an evaluation output.

Every sha256 in the manifest was computed on this disk before upload and each file was verified to exist remotely with a matching byte size.

Companion repositories

ncylich/temporal-moe-ckpts, Megatron training checkpoints
ncylich/temporal-moe-router-adapt, router adaptation safetensors
ncylich/temporal-moe-extras, captures, merged model, result tables, figures
ncylich/temporal-moe-corpus, tokenized training corpus

Provenance

Trained with a personal fork of FLAME-MoE. Not affiliated with or endorsed by the FLAME-MoE authors.

ncylich/temporal-moe-extras