Views
No views yet
(speculative_num_steps, speculative_eagle_topk, speculative_num_draft_tokens) for sglang + EAGLE3 so
GPU energy utilization lands inside a 95-98% band. All trained on the
eagle3-speculative-decoding-energy-sweep
dataset. Full writeup, sweep mechanism, and live-validated results:
project README.[batch_size/8.0, gpu_temp_c/100.0, gpu_mem_used_mb/8192.0, gpu_util_pct/100.0];
output is an index into the same 19-action space (RL/policy.py in the repo
above), decoded to the three sglang flags.| file | algorithm | format | load |
|---|---|---|---|
mlp_bandit/policy.pth | contextual bandit (QNetwork) | torch state_dict | QNetwork(4, 19).load_state_dict(torch.load(...)) |
lookup_table/model.json | per-bs empirical best | JSON | json.load(open(...)) -- {batch_size: {"config": [...], "mean_reward": ...}} |
linucb/model.npz | LinUCB | numpy | np.load(...) -- theta (per-action weight vectors), A (per-action design matrices), actions |
thompson_sampling/model.npz | Bayesian linear TS | numpy | np.load(...) -- mean, cov (per-action posteriors), actions |
gbt/model.joblib | gradient boosted trees | sklearn | joblib.load(...) -- GradientBoostingRegressor, features [state, action_idx/19] |
doubly_robust/model.joblib | doubly robust (direct model) | sklearn | joblib.load(...) -- LinearRegression, IPS correction term is not persisted (recomputed from raw sweep data at pick time) |
cql/policy.pth | conservative Q-learning | torch state_dict | QNetwork(4, 19).load_state_dict(torch.load(...)) |
bcq/policy.pth | discrete BCQ | torch state_dict | torch.load(...) -- dict with behavior_state_dict, q_state_dict, actions |
mlp_bandit, lookup_table, and doubly_robust agree on every batch size
and are the live-validated picks. cql and bcq collapsed to the
non-speculative baseline past bs=1 (overly conservative default
hyperparameters against this reward scale) and are not recommended --
kept here for completeness, not as a suggested pick. See the project README
for the full live A/B numbers per algorithm.unsloth/Llama-3.2-1B-Instruct target +
rescommons/SpecForge-EAGLE3-Llama-3.2-1B-Instruct draft, 80W power cap.
Picks are specific to this hardware/model pair -- retrain on the linked
dataset (or a fresh sweep) before trusting these on different hardware.