5-fold stratified CV on f1_macro, grid [0.1, 1.0, 10.0]
Selected C
10.0
Why not full fine-tuning of DeBERTa-v3 + LoRA (the original CLAUDE.md §7
choice)? See the project's
LITERATURE_REVIEW.md §17:
for short-text small-vocabulary classification with weakly supervised
labels (Peters et al 2019; Tunstall et al 2022, SetFit) a linear probe on
a strong general-purpose embedder reaches the same operating point at
~100x less compute. v2 will revisit fine-tuning on a hand-labeled set.
Rows where the regex matched a specific keyword get the explicit label.
Rows where the regex defaulted (e.g. "mid" for unmatched seniority
titles, "Other" for unmatched role-family titles) are dropped from
training — that fallback signal is too noisy to teach from.
The model's job is to generalize the regex via the encoder's semantic
embedding space — it should classify titles like "ML Researcher" correctly
even though the regex didn't match them. This means:
Eval metrics here measure agreement with the regex on a held-out
slice — they don't measure agreement with a hand-labeled gold standard.
CLAUDE.md §7 calls for a hand-labeled clean test set of 500 examples
for proper evaluation. v2 will land this when capacity allows.
Confidence scores (predict_proba) are useful for filtering high-
confidence predictions in downstream consumers.
1from models.seniority.predict import SeniorityClassifier
2clf = SeniorityClassifier.load_from_hub()3clf.predict(["Senior ML Engineer at Stripe"])
Independent-labeler eval (reviewed gold)
To check the classifier didn't just memorize the regex, we sampled 230
diverse rows per classifier and ran a two-pass Claude labeling protocol:
5 first-pass labelers in parallel (each labeling one shard with
strict taxonomy rules), then 5 second-pass reviewers in parallel
(each shown the first-pass proposal + the trained classifier's
prediction, with default-to-accept plus override criteria for
title-vs-label contradictions). 8/460 rows were overridden by the
reviewers; 1 was skipped as genuinely ambiguous. The resulting test set
lives at eval/seniority_test.jsonl with full provenance per row.
The classifier is scored on the subset of rows whose final label is one
it was trained to predict (117/230 rows — the rest are the
regex-default labels we drop from training, mostly mid).
Metric
All in-vocab
LLM high-confidence subset
n
117
117
accuracy
0.812
0.812
f1_macro
0.8116 (95% CI [0.7347, 0.8729])
0.8116
These numbers come from a different labeler than the training data, so
they're a stronger signal of generalization than the regex-agreement
metric above. Caveats: Claude's labels (even after review) are not human
gold; the in-vocab filter excludes the regex-default rows; the
classifier is a specialist over the explicit labels, not a 9-way
general-purpose classifier.
