QMSum locate-then-summarize: span-trained SegEnc (406M)
A 406M Segment Encoder (Fusion-in-Decoder) fine-tuned to summarize retrieved transcript spans
rather than a full meeting transcript. It is the artifact behind this project's central result:
what a model was trained to read outweighs how large it is.
Handed 2,000-word retrieved spans in place of the long transcript it was built for, the released
Socratic SegEnc checkpoint loses 6.3 ROUGE-1 and falls below our 1.2B system. Fine-tuned for that
span regime under a data recipe verified byte-identical to our own summarizer's, it recovers the
loss entirely and matches our 1.2B on every metric we report, using a third of the parameters and
less than half the peak inference memory, on a sixth of the input it was originally trained on.
This is part of the release for the paper Training regime outweighs scale in query-focused meeting
summarization on QMSum (Ertas AI, in preparation).
The release
Model details
- Base model:
Salesforce/socratic-pretraining-qmsum (Pagnoni et al., ACL 2023), a
question-driven pretrained Segment Encoder already fine-tuned on QMSum
- Architecture: Segment Encoder / Fusion-in-Decoder over BART-large. The transcript is split
into overlapping fixed-length segments, each encoded independently, and the decoder cross-attends
over the concatenated segment encodings
- Parameters: 406,290,432, measured
- Weights:
model.safetensors, 1,625,463,896 bytes, float32
- Training data: the same span-regime targets as our 1.2B summarizer, verified byte-identical,
built from QMSum's own spans and references. No synthetic data is used anywhere in this project
- Chunking:
max_num_chunks 8, which is 4,008 tokens of capacity and covers the 2,000-word
inference budget and the 3,000-word training source without truncation
Training, in two stages
Both stages ran on one NVIDIA RTX 5070 Ti (16 GB), about 45 minutes of GPU time in total.
| Stage | From | Epochs | LR | Warmup | Grad accum |
|---|
| 1 | Salesforce/socratic-pretraining-qmsum | 4 | 3e-5 | 0.1 | 8 |
| 2 | stage 1, epoch 4 | 4 | 1e-5 | 0.03 | 8 |
The released weights are stage 2, epoch 4, which is effective epoch 8. It is the last
checkpoint, chosen with no selection: the validation-best epoch was rejected as selection on noise.
The learning-rate schedule restarts on resume, so this is a continued fine-tune rather than one
clean 8-epoch run. The loss curve for stage 2 ships as history.json.
Results, full official QMSum test split
n=281, greedy decoding, one frozen scorer, a single test touch. Retrieval is identical for both
rows: the promoted locator at 375-word windows, packed to a 2,000-word budget.
| System | Params | R1 | R2 | R-L | R-Lsum | BERTScore | Peak VRAM |
|---|
| Span-trained SegEnc (this model) | 406M | 36.33 | 12.72 | 23.69 | 32.17 | 0.8710 | 2.655 GB |
| Our 1.2B locate-then-summarize | 1.2B | 35.41 | 12.28 | 24.63 | 31.36 | 0.8733 | 5.726 GB |
The difference is +0.93 ROUGE-1 with a 95% paired bootstrap interval of [-0.42, +2.24], which
crosses zero. The two systems are statistically level on quality, and the result is therefore a
cost result rather than a quality one: matched quality at a third of the parameters and less
than half the peak inference memory.
Intended use and limitations
Research use, for reproducing and extending the results in the paper.
- Scope. It summarizes located spans for a query. Given a full untruncated transcript it is
outside the regime it was trained for, which is the entire point of the experiment.
- Metrics. The results above are reference-anchored ROUGE and BERTScore. Concurrent fact-level
evaluation in the paper orders the frontier systems above ours, and that axis is not measured for
this checkpoint.
- Run-to-run variance. Reseeding the 1.2B summarizer moves its ROUGE-1 by 1.59 points, and this
benchmark's detection floor is roughly one ROUGE-1 point at full split size. Read the +0.93 with
both in mind.
- The port caveat. Our port of the authors' inference code sits 3.3 ROUGE-1 below their own
released predictions. Comparisons against the authors' full-input pipeline are scoped accordingly
in the paper; this checkpoint's rows are internally consistent because every one of them runs
through the same port.
Licence and attribution
This model is a modified version of Salesforce/socratic-pretraining-qmsum, fine-tuned by
Ertas AI on retrieved-span data as described above. The base model is distributed under the BSD
3-Clause licence, and this derivative is released under the same licence. The full licence text,
including the conditions and the disclaimer, ships in this repository as LICENSE.
Copyright (c) 2021, Salesforce.com, Inc. All rights reserved.
Salesforce's model card states that its release is "for research purposes only in support of an
academic paper." We record that here alongside the BSD 3-Clause tag the repository carries, and our
own use is academic research accompanying a paper.
Neither the name of Salesforce.com nor the names of its contributors are used to endorse or promote
this derivative. The base model's authors have no involvement in this work.
Underlying data: QMSum is MIT licensed and draws on the AMI and ICSI meeting corpora (CC BY 4.0)
and parliamentary committee proceedings. This repository redistributes no transcript text.
Citation
The paper is in preparation. Until it is posted, please cite the base model's authors alongside any
use of this checkpoint:
1@inproceedings{pagnoni2023socratic,
2 author = {Artidoro Pagnoni and Alexander R. Fabbri and Wojciech Kry\'{s}ci\'{n}ski and Chien-Sheng Wu},
3 title = {Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization},
4 booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics},
5 year = {2023},
6 note = {ACL Anthology 2023.acl-long.713. arXiv:2212.10449},
7}
8
9@inproceedings{zhong2021qmsum,
10 author = {Ming Zhong and Da Yin and Tao Yu and Ahmad Zaidi and Mutethia Mutuma and Rahul Jha and Ahmed Hassan Awadallah and Asli Celikyilmaz and Yang Liu and Xipeng Qiu and Dragomir Radev},
11 title = {{QMSum}: A New Benchmark for Query-based Multi-domain Meeting Summarization},
12 booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
13 pages = {5905-5921},
14 year = {2021},
15 note = {ACL Anthology 2021.naacl-main.472. arXiv:2104.05938},
16}
