You can deploy this model and start earning money today!

akshit-Gupta/
qads-qwen06b-scifact

Name: qads-qwen06b-scifact
Availability: InStock

2602.03306

Views

No views yet

QADS: Query-Aware Adaptive Dimension Selection for Dense Retrieval

Implementation of Learning to Select: Query-Aware Adaptive Dimension Selection for Dense Retrieval on SciFact with Qwen3-Embedding-0.6B.

Paper Target Results (Qwen-0.6B on SciFact)

Method	NDCG@10	Dims Used
Baseline	0.702	100% (1024)
QADS (peak)	0.845	32% (328)
QADS (@30%)	0.839	30% (307)

Method Summary

QADS learns a per-query dimension importance predictor — a single linear layer f_θ: ℝ^D → ℝ^D — that identifies which embedding dimensions are most relevant for each query. At inference, only the top-k dimensions are used for similarity computation, improving retrieval over full-dimensional baselines.

Training pipeline:

Generate L2-normalised embeddings for all queries & documents (frozen Qwen3-Embedding-0.6B)
Compute oracle dimension importance per query using relevance labels + hard negatives
Train a linear predictor to map query embedding → importance distribution via KL divergence
At inference: select top-k dimensions per query, compute masked dot-product similarity

Quick Start

bash

# Requires GPU (T4 or better)
pip install torch transformers>=4.51.0 sentence-transformers datasets trackio huggingface_hub numpy

python qads_scifact.py

Hyperparameters (from paper Appendix A.2)

Parameter	Value
Optimizer	AdamW
Learning rate	1e-4
Weight decay	0.01
Batch size	256
Dropout	0.1
Hard-negative pool K	1000
Sampled negatives M	64
Temperature τ	grid: {0.0001 … 0.05}
Epochs	grid: {20, 30, 50, 75, 100, 200}
Model selection	maximise Σ NDCG@10 at {D, D/2, D/4, D/8} on validation

Files

qads_scifact.py — Complete end-to-end pipeline (embedding → oracle → train → evaluate)
best_predictor.pt — Trained predictor weights (uploaded after training)
results.json — Full evaluation results (uploaded after training)

akshit-Gupta/qads-qwen06b-scifact