Quanfire Government Reranker — rerank-gov-indic (cross-lingual, Indian government press releases)

✅ First defensible number — 2026-08-25 · this checkpoint +88.7 %, three seeded draws span +62 % to +89 %

The re-run this card promised has landed. The previous banner said a run with validation and best-checkpoint selection was queued, and that until it landed no effectiveness number should be quoted. It has landed, seeded, three times.

draw	seed	Recall@1	delta	95 % CI (paired, B=2000)	val loss (selection metric)
1	3001	0.1016	+0.0390 (+62.4 %)	[+0.0271, +0.0519]	0.0768
3 — published here	3003	0.1180	+0.0555 (+88.7 %)	[+0.0426, +0.0688]	0.0570 ← best
4	3004	0.1091	+0.0466 (+74.5 %)	[+0.0333, +0.0590]	0.0764

Every interval excludes 0. All three ran on one evaluation — 2,254 cross-lingual queries, a 1,077-passage pool — with a first-stage baseline of 0.06255546 and a Recall@100 ceiling of 0.92590949, identical to eight decimals in all three. Those invariants are how you can tell the draws differ by seed and by nothing else.

Read the range, not the headline. Retraining this procedure yourself should land somewhere in +62 % to +89 % (a 1.42× spread). From these weights, the delta in the table.

A coincidence we are pointing at rather than hiding. The published checkpoint was chosen on validation loss — carved from training documents, never touching the evaluation set — and it also happens to be the highest test result. That will look like cherry-picking, so: at n=3 the two coinciding is about chance, and our sibling rerank-statute-en is the counter-example, where validation picked the third-best of four draws. Across all seven seeded draws in this family, validation loss shows no consistent relationship with test recall. The protocol is what makes the number honest, not the outcome of the protocol.

What this fixes, and it is the reason the old numbers were void. The runners behind the two withdrawn figures seeded their split, their negative mining and their validation carve — and seeded nothing in training, so the shuffle order and the freshly-initialised ranking head came from OS entropy on every run, under an effective batch size of 1. That family was demonstrably bimodal: a second semi-hard draw on byte-identical mined negatives selected a checkpoint at chance (best val 1.9198 against ln 7 = 1.9459) where the first reached 0.0560. This runner calls random.seed, torch.manual_seed and torch.cuda.manual_seed_all after the split and mining and before the head is constructed. Three seeded draws produced no null, no sign change and nothing near chance.

Neither withdrawn figure is corrected by this. +31.9 % was measured on a contaminated split and measures nothing; +97.2 % had a clean split but scored last-step weights. This is reported from zero, on its own split, and must not be differenced against either.

Seeded is not bit-reproducible. No deterministic-algorithm flag is set and no same-seed re-run has been attempted. The draws differ by seed by design rather than by uncontrolled entropy; that re-running seed 3003 returns 0.1180 is not established and is not claimed.

A fourth draw was attempted and did not complete. All three of its permitted launches wedged before writing a single byte — a pre-CUDA hang on this box that cannot be signalled — so it produced no number of any kind. Recorded because "one draw is missing" invites the suspicion that an inconvenient result was dropped: there was none to drop. Missing data, not a filtered set.

The account of the original contamination and the two withdrawals follows, unchanged.

The original figure was measured on a split built by a plain shuffle — random.shuffle(rows), first 1,200 rows to eval, the rest to train — with no exclusion of any kind. Replaying it showed 96.4 % of held-out queries came from a press release that was also in training and only 42 (3.5 %) were genuinely unseen. For a cross-lingual claim that is fatal: the model had already been shown almost every answer in another language.

The re-measure holds out whole source documents (264 of 1,505), asserts before training that no held-out positive, query or document appears in training, and draws the candidate pool to 1,077 unique passages so it is comparable to the original run's 1,096. Training volume falls only from 10,269 rows to 9,200 (89.6 %) — this is the one figure in the family whose clean and contaminated versions are close to volume-matched, so the change is mostly the split and not the data budget.

What changed, honestly. The relative gain went up and the absolute numbers collapsed, and both have the same cause: on press releases it has never seen, the first-stage retriever falls from 0.3083 to 0.0626. A reranker helps most when the first stage is weak, so a larger percentage over a much weaker baseline is not this model getting better. The number that deflates hardest is the one worth reading: of the queries whose answer is anywhere in the top-100, the reranker lifts 13.3 % to rank 1, against the 42.9 % this card used to claim.

What is unchanged. The weights, and the corpus fact underneath the curve: the retriever's top ranks really are dense with near-duplicate releases about the same event, and that is now measured rather than asserted (see Training). What is NOT unchanged, corrected 2026-08-23: the negative-hardness ordering does not survive. An earlier version of this paragraph said it did, on the grounds that the three arms were measured identically against each other. They were — but each was measured once, and training in these runs was never seeded. Re-running the shipped arm on the same split with byte-identical mined negatives returned −68.8 % where the first run returned +106.4 %: non-overlapping intervals, opposite signs. The spread within that one arm is 95.7 % of the spread between the best and worst arms, so an arm effect is not separable from a run-to-run effect here. No percentage on this card is withdrawn by that — what is withdrawn is the ordering they were arranged into.

A correction of record: an earlier internal notice attributed this defect to the shared fine-tuning pipeline's split filter. That was wrong — this script never used that pipeline. The defect here was its own, and simpler.

It is a reranker, not a retriever. It does not produce embeddings and cannot be searched with cosine similarity. It takes a query and a shortlist of candidate passages (produced by a bi-encoder such as embed-gov-indic) and re-scores each pair, moving the right passage up the list. Use it on top of a retriever, never instead of one.

• Framework & code: github.com/quanfire-ai/quanfire-multilingual-embedding (Apache-2.0)
• Weights licence: Apache-2.0. The reranker is ours; the training text is PIB press-release content reused under PIB's reproduction policy (royalty-free reproduction with attribution; by our four-gate reading, no NonCommercial and no ShareAlike constraints apply). The model is non-reconstructive — it emits a relevance score, never the source text.
• Pairs with: embed-gov-indic as the first-stage retriever (the reranker was trained and evaluated against that retriever's own candidate lists).

What it is for

Measured result

• Delta +0.0608 (+97.2 %), paired 95 % CI [+0.0475, +0.0745] — the interval excludes 0. WITHDRAWN 2026-08-23 — scored on last-step weights, see the banner. The confidence interval was computed correctly; it is an interval around the wrong checkpoint's score. The reranker roughly doubles the rate at which the correct passage is ranked first.
• Of the queries whose gold passage is anywhere in the retriever's top-100 (a 0.9259 ceiling), the reranker lifts 13.3 % of them to #1. Most of the headroom is unspent.
• The task is genuinely cross-lingual: only 6.8 % of top-100 candidates are in the same language as the gold passage, so the model is not solving this by language matching.

Note on the first-stage figure. The embed-gov-indic Recall@1 shown here (0.0626) is measured under this evaluation's regime — empty prefixes, a 1,077-passage pool, this specific 2,254-query cross-lingual sample, and a document-isolated split. It is not the same slice as the headline Recall@1 on the embed-gov-indic card (0.2518 on its own clean split) and will differ from it. The retriever adapter itself was trained on a contaminated split; scoring it on held-out documents is what produces the low figure here.

We report disjoint / excludes-0 confidence intervals, and deliberately avoid the phrase "statistically significant."

Which weights should I use?

Scope — what is and isn't validated

How to use

Training

• Base: intfloat/multilingual-e5-small (MIT), full fine-tune with a single-logit sequence-classification head; max_length 256; 2 epochs, lr 2e-5 (bf16, CUDA). Data seeding only, corrected 2026-08-23. This line previously read seed 0. That is true of the split, the negative mining and the bootstrap — a re-run reproduced the mining ledger byte-for-byte — and false of training, which is what the phrase implies here. The training order uses an unseeded global RNG and the ranking head is randomly initialised, so these weights are one draw and are not bit-reproducible.
• Data: PIB press releases — the same release published in many languages, giving naturally-parallel cross-lingual pairs. Same corpus family as embed-gov-indic.
• Negatives — semi-hard, and this mattered: for each query, the negatives are drawn from the first-stage retriever's own candidate ranks [20, 100) — hard enough to teach real cross-lingual discrimination, but past the near-duplicate head. Two guards: the gold passage is excluded, and any candidate that is a parallel translation of the query's own press release (shares its source document) is excluded as a false negative.
• The negative-hardness curve is why. We ran three single-variable versions and the choice of negative made or broke the model:
  • random global negatives (too easy): +10.5% — a real but thin gain;
  • the retriever's hardest top candidates (too hard): −45.1% — a regression. Those top ranks are dominated by near-duplicate releases about the same event, so treating them as negatives teaches the model to demote correct answers;
  • semi-hard negatives from ranks [20, 100) (this model): +31.9%, and its clean-split re-measure +97.2 % is withdrawn too — both void, for different reasons.
  All three of the original percentages are withdrawn (2026-08-21) — the three arms shared the contaminated split described at the top of this card. Only the shipped arm was ever re-measured, and that re-measure is itself now withdrawn (2026-08-23, last-step weights).
  The ordering is withdrawn too, 2026-08-23. This card used to say the ordering survived because the three arms were measured identically against each other. They were — and each was measured once, from a training loop that is not seeded. The outstanding work named here (re-running the other arms on the document-isolated split) was done, and then the shipped arm itself was re-run: on the same split, with a byte-identical mining ledger and an exact retriever-side control, it returned −68.8 % against the first run's +106.4 % — non-overlapping intervals on opposite sides of zero. The spread within that single arm is 95.7 % of the spread between the best and worst arms, and the second draw landed beside the hardest-negative arm. A regression the size and sign of the hardest-negative arm's was produced by an arm that uses no hardest-rank negatives at all, so this evaluation does not support hardest-rank negatives cause the regression.
  So: “non-monotonic”, “sweet spot”, “strictly dominates”, “a real but thin gain” and the cliff are all withdrawn as findings of this evaluation. Not because any measurement was wrong, but because single draws from an unseeded procedure cannot be ordered. If you want the ordering for your own corpus, the instrument is n seeded draws per arm with each seed recorded — fix n before the first draw.
  What did survive is the corpus fact, and it got sharper, because mining IS seeded. Over a full untruncated scan of the retriever's head — 8,216 anchors × top-160 = 1,314,560 candidates — parallel translations of the query's own press release are 4.30 % of the top-160 (56,529) and ≥7.76 % of the top-64, against 0.201 % of uniform draws from the pool: at least 38.6× denser at the head of the ranking, with ≥72.2 % of all top-160 contamination inside the first 64 ranks. Both head figures are floors. This describes what the retriever returns. What it does to a trained reranker is exactly what this evaluation can no longer tell you.

Limitations

• Ceiling-bound by the retriever. The reranker can only reorder what the first stage returns; a gold passage absent from the shortlist is unrecoverable.
• Domain specialist. Government press-release text only; no cross-domain validation.
• Single-gold evaluation understates it, by an unmeasured amount. The corpus contains many near-duplicate releases about the same event across ministries and dates; when the reranker ranks such a near-duplicate first it counts as a miss. The direction is known, the magnitude is not, so treat it as a caveat rather than a discount to apply.
• Lowest-resource languages are thin in the underlying corpus (e.g. Khasi, Nepali, Manipuri); treat any implied per-language behaviour there as indicative only.
• Absolute performance is low. On press releases it has never seen, the full retrieve-then-rerank pipeline puts the right passage first about 12 % of the time, and converts only 13.3 % of recoverable queries to rank 1. The reranker's contribution is real and measured; the pipeline it sits in is not yet strong. Measure it on your own data before depending on it.
• The first stage is the binding constraint. Most of the loss is the retriever failing on unseen documents (0.0626), not the reranker failing to reorder. The 0.9259 top-100 ceiling says the answers are mostly there; converting them is where the remaining work is, and a stronger first stage would move this more than a stronger reranker.
• Training was never seeded, so every number here is one draw. Updated 2026-08-23: all three arms were re-measured on the clean split, and then the shipped arm was drawn a second time and disagreed with itself by more than the arms disagree with each other. Each magnitude on this card is a valid measurement of its checkpoint and none is a measurement of the recipe. Treat any comparison between two of them as unestablished.
• No revision of this repository has a valid score. main holds the contaminated-split weights and clean-2026-08-21 holds the last-step weights; both of their numbers are withdrawn. The only checkpoint that ever produced a defensible number was never uploaded. This is stated plainly so nobody has to infer it from the strikethroughs above.

Attribution

Citation