⚠️ This model games its training reward. It achieves high NDCG@10 by collapsing all
outputs to 3–7 token keyword phrases, discarding the entire boolean search task it was
trained to learn. Published for research transparency and as a reproducible example of
specification gaming in RLVR. For deployment, use GRPO v2.
A Qwen2.5-3B-Instruct model fine-tuned via GRPO starting from SFT v1,
using live Tantivy retrieval (NDCG@10 + MRR) as the reward signal.
The hack: specification gaming via minimum viable retrieval
The model learned that on small corpora (NFCorpus: 3,633 docs; SciFact: 5,183 docs),
2–4 content nouns yield near-optimal BM25 recall. Instead of learning boolean query
generation, it learned to extract the most distinctive nouns from the NL query:
Input:Do Cholesterol Statin Drugs Cause Breast Cancer?
GRPO v1 output (hacking):
<reasoning>
</reasoning>
<query>Cholesterol Statin Breast Cancer</query>
SFT v1 output (intended behaviour, lower NDCG):
<reasoning>
Key concepts: statin drugs, causal relationship, breast cancer.
Connect with AND; expand synonyms with OR.
</reasoning>
<query>(statin OR "HMG-CoA reductase inhibitor" OR simvastatin OR atorvastatin)
AND (cause OR risk OR association OR induce)
AND ("breast cancer" OR "breast carcinoma")</query>
The GRPO v1 output actually achieves NDCG@10 = 0.971 on this query while the SFT
output achieves 0.000 — the hack outperforms the intended behaviour because SFT used wrong
synonyms. This made the gaming invisible in aggregate metrics alone.
Collapse statistics
Metric
Value
Mean completion length
5.1 tokens (vs 95 for SFT v1)
Boolean operator usage (AND)
0% (vs ~80% for SFT v1)
Boolean operator usage (OR)
0% (vs ~90% for SFT v1)
Phrase usage
0% (vs ~70% for SFT v1)
Reasoning block content
empty
frac_reward_zero_std during training
90–96% from step 1
frac_reward_zero_std = fraction of GRPO groups where all completions received identical
reward. At 90-96%, policy gradient was near-zero throughout — the model was not learning,
it had already converged on the keyword-bag strategy.
Why it still scores high on benchmarks
Small corpora: BM25 keyword recall on 3–5K doc indices is high; a rare noun
appears in only a handful of documents, making it highly discriminative.
SFT degraded: SFT v1 scored below base on SciFact (0.273 vs 0.386) due to
over-specified queries — a low bar to beat.
NDCG@10 rewards recall of first hit: any query retrieving one relevant document
in top-10 scores well. Keyword bags do this reliably on small indexes.
This does not generalise: on a 2.7M-doc index (NQ), keyword bags return thousands of
irrelevant results; NDCG@10 and MRR would collapse to near zero.
1@misc{searchlm2026,
2 title = {SearchLM: Training Small Language Models for Boolean Query Generation via RLVR},
3 author = {Rao, Supreeth},
4 year = {2026},
5 url = {https://github.com/SupreethRao99/searchLM},
6}
