You can deploy this model and start earning money today!

NagaYu/
flashback-first-bad-step

Name: flashback-first-bad-step
Rating: 2.5 (12 reviews)

Views

No views yet

Flashback: first-bad-step detector

Given a run's sketch index -- a few hundred bytes of summary statistics per training step -- this model reports the step at which the run first went wrong.

It is small on purpose: 10 logistic weights plus 9 fitted consensus hyperparameters. The interesting part is not the model, it is that the input is cheap enough to record for every step of every run.

Results on held-out runs

detector	mean abs. error (steps)	exact	within 2	never detected
loss curve alone	0.50	2/6	4/6	2
this model	0.00	6/6	6/6	0

Train split: 12 runs, MAE 0.00. Splits are by run: no step of a test run appears in training.

Usage

python

from flashback.detector import FirstBadStepDetector
from flashback.sketch import StepSketchIndex
from huggingface_hub import snapshot_download

det = FirstBadStepDetector.load(snapshot_download("<user>/flashback-first-bad-step"))
report = det.predict(StepSketchIndex("runs/my-run/sketch", mode="r"))
print(report.first_bad_step, report.confidence, report.top_metrics)

To produce the sketch index in the first place, add three lines to your training loop:

python

from flashback.integrations import FlashbackCallback
trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")])

Fitted configuration

json

{
  "win": 32,
  "gap": 1,
  "mode": "diff",
  "k_cal": 8.0,
  "k_min": 8.0,
  "min_votes": 3,
  "vote_window": 2,
  "exclude": [
    "lr",
    "step_wall_ms",
    "batch_id_mean",
    "batch_uniq_frac"
  ],
  "use_logistic": true,
  "logistic_threshold": 0.5,
  "version": "1"
}

Feature weights

feature	weight
`bias`	-9.8951
`log1p_max_z`	+0.1471
`log1p_top5_mean_z`	+0.1950
`frac_metrics_firing`	+18.4084
`hard_signal`	+1.2226
`log1p_z_grad_norm`	+0.3008
`log1p_z_update_max`	+0.1584
`log1p_z_loss`	+0.0378
`log1p_z_hist_tail`	+0.2497
`rel_position`	+15.0875

Limitations

It needs a healthy stretch of the run to calibrate against; a run that is broken from step 0 has no baseline and will not be localised.
It reports regime changes, so a fault that ramps in smoothly over hundreds of steps has no single correct answer and the reported step will be early or late by roughly the ramp length.
It was fitted on small Transformers (a few hundred thousand to a few million parameters). The features are scale-free by construction, but transfer to very large models is not demonstrated here.
No claim of bitwise determinism is made anywhere in Flashback; this is a statistical detector over recorded statistics.

Generated by Flashback v0.1.0.

NagaYu/flashback-first-bad-step