jobbert-ner-sonnet-v1
Distilled Named Entity Recognition model for English-language job postings. One of six students produced for the paper Distributed NER on Spark: A Teacher-Student Pipeline for Large-Scale Entity Extraction from Job Postings (Soltani and Hanine 2026).
- Teacher: Claude Sonnet 4.6 (labels acquired via AWS Bedrock)
- Architecture: jjzha/jobbert-base-cased fine-tuned for 8-class token classification
- Student identifier:
s3_jobbert_sonnet
- Artefact size: ~420 MB
Latency is materially lower than the spaCy students (~16 ms vs ~44 ms per document on the gold set). F1 is materially lower on the full distribution because the 512-token window cannot reach the trailing part of most postings; on shorter text the model recovers the full entity set.
Intended use
Entity extraction from English-language job-posting descriptions into an eight-type schema:
SKILL, JOB_TITLE, COMPANY, LOCATION, EXPERIENCE_LEVEL, EDUCATION, CERT, COMPENSATION.
Appropriate downstream applications include posting indexing for search and analytics, skill-demand aggregation for labour-market research, cost-quality-speed benchmarking of distilled NER, and teaching use in NLP / distillation courses.
Out-of-scope use
Not suitable for:
- CVs or résumés (different register; a CV-trained model should be used instead).
- Non-English postings.
- Fully-automated candidate screening or hiring decisions; downstream ranking or filtering should be built only after an application-side schema and bias review (see Ethical considerations).
- Medical, legal, financial or other high-stakes decision support.
- Posting text from languages or locales for which the underlying teacher labels were not representative.
Training
- Teacher labels: 5,000 stratified postings labelled by Claude Sonnet 4.6 in a single run at temperature 0.
max_tokens was raised from 4,096 to 8,192 mid-run after two truncation failures on entity-dense postings; final labels from the fixed-ceiling run were used.
- Curator: 80/10/10 train/dev/test split by
md5(job_link) mod 10, so Sonnet- and Haiku-trained students see the same posting partitions.
- Hardware: one NVIDIA A10G 24 GB GPU (AWS g5.xlarge).
- Training seed: 42.
- Principal hyperparameters and full training spec:
pipeline/training/experiments/specs/s3_jobbert_sonnet.yaml in the accompanying project repository.
Evaluation
Sonnet-trained students evaluate on all 516 gold postings; Haiku-trained students evaluate on 515 because one posting was dropped by the curator for zero-entity teacher output during the Haiku run. Metric: micro-F1 over exact (text, type) tuples; character-offset matching is relaxed. Entities are deduplicated within a posting before comparison.
| Overall | Value |
|---|
| Micro-F1 | 0.2811 |
| Precision | 0.3397 |
| Recall | 0.2398 |
| 95% CI | [0.273, 0.289] (entity-level delta method) |
| Latency mean (eval hardware) | 15.66 ms / document |
| Latency p99 (eval hardware) | 18.1 ms / document |
| Text coverage | first 512 BERT tokens (~first third of an average posting at 3,996 characters mean length) |
| Postings evaluated | 516 (of the 516-posting gold set) |
Per-entity-type
Per-entity numbers below reflect the coverage constraint as much as the model's per-type quality. Entities that appear only in the trailing part of a long posting (typically CERT, EDUCATION, EXPERIENCE_LEVEL, and COMPENSATION in many English templates) are systematically outside the model's input window and therefore missed at the recall metric even when the model would classify them correctly on shorter text. For full-text coverage, use the spaCy variant.
| Entity type | P | R | F1 |
|---|
| COMPANY | 0.673 | 0.478 | 0.559 |
| JOB_TITLE | 0.614 | 0.513 | 0.559 |
| LOCATION | 0.569 | 0.475 | 0.518 |
| COMPENSATION | 0.240 | 0.070 | 0.108 |
| EDUCATION | 0.127 | 0.021 | 0.035 |
| CERT | 0.259 | 0.072 | 0.112 |
| EXPERIENCE_LEVEL | 0.071 | 0.016 | 0.026 |
| SKILL | 0.078 | 0.116 | 0.093 |
Teacher comparison
The teacher (Claude Sonnet 4.6) reaches micro-F1 = 0.5171 against the same gold set (95% bootstrap CI [0.503, 0.530]). The student trails the teacher by 0.236 points absolute (45.6% relative). See paper §4.3 for the full comparison and the error-mode analysis of this student's residuals.
Usage
1from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
2
3tokenizer = AutoTokenizer.from_pretrained("AchrafSoltani/jobbert-ner-sonnet-v1")
4model = AutoModelForTokenClassification.from_pretrained("AchrafSoltani/jobbert-ner-sonnet-v1")
5ner = pipeline("token-classification", model=model, tokenizer=tokenizer,
6 aggregation_strategy="simple")
7
8text = 'Senior Machine Learning Engineer at Acme Corp in Berlin. Requires 5+ years of experience with PyTorch, AWS, and Kubernetes. MSc in Computer Science preferred. Salary $140,000 – $180,000.'
9for ent in ner(text):
10 print(ent["word"], "->", ent["entity_group"])
11
12# Produces (verified on this release; note the BERT wordpiece tokenisation
13# artefacts in numeric spans):
14# Senior Machine Learning Engineer -> JOB_TITLE (0.79)
15# Acme Corp -> COMPANY (0.87)
16# Berlin -> LOCATION (0.69)
17# 5 + years -> EXPERIENCE_LEVEL (0.85)
18# PyTorch -> SKILL (0.75)
19# AWS -> SKILL (0.76)
20# Kubernetes -> SKILL (0.79)
21# MSc in Computer Science -> EDUCATION (0.76)
22# $ 140, 000 – $ 180, 000 -> COMPENSATION (0.95)
23
24# Note: the BERT base tokeniser has a 512-token window. The example text here
25# is short (well under the limit); on a full job posting of ~4,000 characters,
26# the tail is truncated and entities there are systematically missed. For
27# full-text coverage, use the spaCy variant.
Ethical considerations
This model extracts entities from job postings, a document class whose downstream consumers are typically hiring, ranking, or matching systems. Three cautions are transplanted from paper §6:
- Schema-induced bias. SKILL over-extraction is inherited from the LLM teacher; soft-skill phrases ("communication skills", "interpersonal skills") and generic tools ("Excel", "CRM") are over-represented relative to a tighter gold standard. A downstream ranker that treats such phrases as filters is encoding the teacher's lexical habits as a hiring criterion and is not recommended without a schema review at the application layer.
- Contested ground truth. A vendor benchmark in the paper against LinkedIn's own
job_skills.csv on 938,028 jointly-present postings yielded 9.56% agreement and 56.44% discovery: the two extraction schemas produce largely non-overlapping views of the same corpus. Neither constitutes a ground truth; the numbers measure schema divergence, not model quality.
- Consent and licensing. The training corpus is a publicly-released Kaggle redistribution of scraped LinkedIn postings. Individuals named in postings (recruiters, hiring managers) did not consent to having their role descriptions re-processed for research. The model is licensed CC BY-NC 4.0 for research and non-commercial evaluation only; any commercial deployment requires a separate legal and ethical review against the data-provenance chain.
Limitations
- Trained and evaluated on English-language LinkedIn postings from a publicly-released 2024 Kaggle redistribution; generalisation to other platforms (Indeed, Stack Overflow, regional job boards) or other languages is unevaluated.
- Gold set is single-annotator (516 postings). Intra-annotator stability was scheduled to be measured one week after the main annotation pass; users should treat the reported F1 as having an un-quantified annotator-noise floor until that number lands.
- Output schema is locked to the eight types above. Finer-grained or taxonomy-aligned schemas require re-training against new labels.
- The underlying BERT tokeniser has a 512-token window; the average gold-set posting is 3,996 characters, so approximately the first third of a typical posting is in context. Entities that appear only in the tail (often qualifications, certifications, benefits) are systematically missed. For full-text coverage, prefer the spaCy variant.
Citation
1@unpublished{soltani2026distilledner,
2 author = {Achraf Soltani and Mohamed Hanine},
3 title = {Distributed NER on Spark: A Teacher-Student Pipeline for Large-Scale Entity Extraction from Job Postings},
4 year = {2026},
5 note = {Advisor: Prof.\ Hanine Mohamed},
6 url = {https://github.com/achrafsoltani/distributed-ner-on-spark},
7}
Licence
- Model weights: CC BY-NC 4.0 — research and non-commercial evaluation only.
- Source code in the accompanying repository: Apache 2.0.
