The -r matters. The pool index is cached for an hour, and a stale index does
not look like an error - it looks like "this run has no checkpoints".
This artifact
run
m4b_klr_pi_s40
global step
280
base model
Qwen3-4B-Base
architecture
Qwen3ForCausalLM, 36 layers, hidden 2560
sharding
verl FSDP, world_size 2
files
14
size
17.7 GB
archived
2026-08-18
integrity
all 14 files verified against the Hub by content hash
What survived, and what did not
Established by direct inspection of the files, not inferred from their sizes:
file
state
evidence
model_world_size_2_rank_{0,1}.pt
intact
parse as ZIP; all 406 members pass CRC; both exactly 8,823,082,523 bytes
extra_state_world_size_2_rank_{0,1}.pt
intact
parse as ZIP; equal sizes
optim_world_size_2_rank_{0,1}.pt
destroyed
BadZipFile: File is not a zip file - the ZIP central directory is written last and never landed. Sizes also disagree between ranks (8,732,745,728 vs 8,778,358,784), impossible for equal FSDP shards
huggingface/*.safetensors
incomplete
shard 1 of 4 only, no model.safetensors.index.json, plus a 4.9 GB .tmp shard caught mid-write
Only the intact files are in this repo. The destroyed optimizer state and the
partial export were deliberately left out rather than uploaded as if they were
data.
Consequence: usable weights - for evaluation, analysis, or initialising
another run - but no optimizer state, so training cannot resume from them.
The local copy this was salvaged from has since been deleted, so this repo is now
the only place these weights exist.
Inside the study's own tooling (private repo yurun-yuan/async-rl) the whole
procedure is one command, which also restores the run-state files and writes the
tracker:
One thing to know if you restore by hand: the trainer discovers a checkpoint
only through latest_checkpointed_iteration.txt in the run directory, and
that file must be written last, after the weights are fully in place. Writing
it early is how a half-transferred checkpoint gets picked up.
The whole study
Every checkpoint from this study is archived the same way, one repo per step:
run
base model
steps archived
m4b_grpo_s40
Qwen3-4B
200, 225
m4b_grpoklr_s40
Qwen3-4B
275
m4b_klrneg_s40
Qwen3-4B
310
m4b_klr_pi_s40
Qwen3-4B
275 (plus step 280, weights only - see below)
m4b_reinforce_s40
Qwen3-4B
50, 75
m4b_klr_rollout_s40
Qwen3-4B
none
m17b_klr_pi_s1024
Qwen3-1.7B
4
m4b_klr_rollout_s40 has no checkpoints anywhere - this is expected, not a
gap in the archive. The trainer's own retention policy had already pruned every
step before the archive was made, so there was never anything to upload. Its
metrics and logs survive; its weights do not.
m4b_klr_pi_s40 step 280 exists only as
RyanYr/asyncrl-partial-m4b_klr_pi_s40-gs280:
the run died mid-save, so it has intact model weights but no optimizer state
and cannot be resumed from. It is deliberately outside the
asyncrl-<run>-gs<step> naming scheme so automated restores can never select it.
To continue that run, use step 275.
