vllm-ltr-predictors — output-length predictors for LTR LLM-inference scheduling
Self-trained output-length predictors for Learning-to-Rank (LTR) scheduling in vLLM, plus the
baseline latency measurements. Capstone artifact (CSCI 6806, FDU; advisor Prof. Jeeho Ryoo),
reproducing & extending vllm-ltr (Fu et al., NeurIPS'24) and testing PARS (Tao et al., 2025,
arXiv:2510.03243).
A predictor estimates a request's output length so the scheduler can approximate Shortest-Job-First,
cutting head-of-line blocking. All numbers below are our own measurements — nothing fabricated.
Predictors (predictors/)
| dir | backbone | loss | role | Kendall's Tau (in-dist) |
|---|
listMLE-opt125m | OPT-125M | listMLE | LTR baseline (reproduced) | 0.559 |
classification-opt125m | OPT-125M | cross-entropy (10 buckets) | ranking-vs-classification control | 0.194 |
PARS-bert | BERT-base | margin(1.0)+δ(0.2) | optimization (ours) | 0.596 |
A1-opt125m-margin | OPT-125M | margin, δ=0.2 | ablation: isolate the loss | 0.543 |
A2-bert-nofilter | BERT-base | margin, δ=0 | ablation: isolate the backbone | 0.598 |
Each dir holds finetuned/model.safetensors + finetuned/config.json + usage_config.json.
Key results
- Baseline (FCFS vs LTR,
results/): at high load LTR cuts TTFT ~2.86× (rate 16: 17.3 s → 6.0 s);
honest cost: per-token latency (p99 TPOT) is worse — the SJF trade-off.
- Ranking ≫ classification: Tau 0.559/0.596 vs 0.194 — confirms the base paper's design choice.
- Generalization (PARS vs listMLE): cross-distribution (ShareGPT) Tau 0.315 → 0.361 (+15% rel.).
- Ablation (honest negative): the cross-dist gain is driven almost entirely by the BERT backbone (+0.065);
the margin loss (−0.012) and δ-filter (−0.007) did not help on this single-GPU 8B setup.
Limitation (disclosed)
The classification predictor trained fine (Tau 0.194) but its end-to-end latency could not be measured:
the fork's tpt-class10 schedule path returns 0 completions across all request rates (a serving bug,
not a config artifact). No classification latency numbers were produced or fabricated.
Base models & data (links only — not redistributed here)
- Serving model:
meta-llama/Meta-Llama-3-8B-Instruct (gated, Meta license)
- Backbones:
facebook/opt-125m (non-commercial research only), google-bert/bert-base-uncased (Apache-2.0)
- In-dist / training data:
lmsys/lmsys-chat-1m (gated)
- Cross-dist data: ShareGPT (community mirrors, e.g.
anon8231489123/ShareGPT_Vicuna_unfiltered; provenance varies)
Derived length-labelled traces are not redistributed (upstream terms); regenerate them from the sources.
License
Mixed: the OPT-125M-derived weights (listMLE, classification, A1) are research / non-commercial only
(inherit OPT's terms); the BERT-derived weights (PARS, A2) are Apache-2.0. Use for academic research only.
Reproduction & citation
Code + reproduction guides:
https://github.com/TaliesinYang/vllm-ltr-optimization
(
docs/REPRODUCE-BASELINE.md,
docs/REPRODUCE-PREDICTORS.md).
- Fu et al., Efficient LLM Scheduling by Learning to Rank, NeurIPS 2024.
- Tao et al., PARS (pairwise margin ranking), 2025, arXiv:2510.03243.
