The ranking model from the Redrob Intelligent Candidate Discovery and Ranking Challenge submission.
Given a 22-feature vector describing a candidate's fit against a job description, this model outputs a relevance score. It is trained on labels generated by 2,500 pairwise judgments from a local LLM (Gemma3) rather than hand-coded heuristics, specifically to avoid label circularity.
This model is one stage inside a larger offline candidate-ranking pipeline. It does not do retrieval, does not compute the input features itself, and does not produce the final rank on its own.
raw_score = this_model.predict(feature_vector)
final_score = raw_score * consistency_score # applied by the host pipeline, not this model
consistency_score is a separate, multiplicative honeypot/data-integrity check computed by the host application — it is not part of this model's output.
Input: a 22-feature float32 vector per candidate (exact feature order below)
Output: a single raw relevance score (higher = more relevant). Not yet penalized for data-integrity issues — combine with a consistency_score downstream before final ranking.
Feature vector (in order)
#
Feature
Formula / Source
1
bm25_score
Stage 1 BM25 retrieval score (normalised)
2
yoe
profile.years_of_experience
3
Param_A_Systems_Depth
Fraction of career months in roles whose descriptions contain retrieval, search, or ranking keywords
Training labels — Gemma3 pairwise annotation (the key differentiator):
Rather than a pure heuristic label, training labels are generated via 2,500 pairwise LLM comparisons using Gemma3:4b-it-q4_K_M running locally on Ollama, with zero external API calls and full reproducibility. A stratified sample of 500 candidates is drawn across three strata (top-100, boundary 101–300, and a broader pool with guaranteed low-consistency coverage), and each candidate receives roughly five matchups against random opponents.
For each pair, Gemma3 reads both candidates' full structured profiles alongside the JD requirements and disqualifiers, then produces a single verdict: CANDIDATE_A, CANDIDATE_B, or TIE. Win and loss tallies convert to Elo ratings via Laplace-smoothed win rates:
Elo ratings are thresholded to 0–3 relevance labels by quartile, producing a balanced training set with roughly 125 candidates per label.
Why this breaks circularity: Gemma had no knowledge of the 22 engineered features, the BM25 scores, or the penalty weights. It learned independently that IR-specific skills (FAISS, BM25, Qdrant, Sentence Transformers) outrank generic ML skills, and that production-company backgrounds outrank consulting-only careers. LightGBM then learns how the 22 features correlate with these independent judgments, surfacing interactions that were never explicitly encoded.
Model Comparison: Heuristic vs. Gemma-Trained
The competition provides no ground-truth relevance labels, so a standard NDCG@10 ablation against a labeled holdout set isn't possible to compute honestly. What is available, and what is reported here, is a direct head-to-head comparison between a LightGBM model trained on the original heuristic weak label and this model (trained on Gemma3 pairwise labels), run on the same candidate pool with the same feature vectors.
Method: both trained models score the full ~8,500-candidate retrieval pool. The same post-inference consistency multiplier is applied to both before ranking, so the comparison isolates the effect of the training label, not the honeypot suppression layer.
Metric
Result
Top-10 overlap between the two models
0 of 10 candidates in common
Spearman rank correlation (top-100)
0.001 — statistically independent rankings
Honeypot leakage, heuristic-trained model
Required a hand-coded post-processing suppression list to keep keyword-stuffed non-technical profiles out of the top 100
Honeypot leakage, Gemma-trained model (this model)
0 of 100 candidates with consistency_score < 0.25, achieved with no post-processing suppression list
Qualitative before/after: prior to the Gemma retrain, the heuristic-trained model's unsuppressed top-10 surfaced profiles such as Content Writer, Project Manager, and Sales Executive — each with AI-sounding skills listed but no underlying technical career history, because the heuristic label rewarded keyword coverage directly. After the Gemma retrain, the same pool's top-10 surfaced candidates with FAISS, BM25, Qdrant, Sentence Transformers, and Hugging Face Transformers in their skill history — sourced from a model that never saw bm25_score or hard_req_coverage during label generation and discovered the IR-relevance ordering independently from reading full candidate profiles.
The two models disagreeing almost completely (Spearman 0.001) is itself evidence of non-circularity: a model trained on labels derived from the same 22 features it predicts on would be expected to correlate strongly with a heuristic built from those same features, not diverge from it entirely.
Intended Use & Limitations
Built for a hackathon submission (Redrob Intelligent Candidate Discovery and Ranking Challenge); not validated for production hiring decisions.
Expects the exact 22-feature schema above, computed by the host pipeline's src/features.py. Feeding hand-built or differently-ordered features will produce meaningless scores.
Raw model output is not the final ranking score — it must be multiplied by a separately computed consistency_score before use.
Trained on a synthetic/competition candidate dataset; label distribution and feature semantics may not generalize to other candidate pools without retraining.
AI Tool Disclosure
Gemma3:4b-it-q4_K_M (Google DeepMind, running locally via Ollama) was used offline to generate 2,500 pairwise relevance judgments on a stratified sample of 500 candidates. These judgments served as independent, non-circular training labels for this model. No candidate data was transmitted to any external service at any point.
