100.0% win rate against the base checkpoint it was fine-tuned from (68W 0L 0T), judged blind. Against Qwen2.5-1.5B-Instruct — a real instruction-following model that was never our training baseline — 76.5%.

Results

vs base

Prompt set	n	Win rate	W
Overall	68	100.0%	68
Held-out domain tasks	40	100.0%	40
Out-of-template probes	21	100.0%	21
False-premise prompts	7	100.0%	7

vs instruct

Prompt set	n	Win rate	W	L	T
Overall	68	76.5%	43	7	18
Held-out domain tasks	40	92.5%	34	0	6
Out-of-template probes	21	45.2%	4	6	11
False-premise prompts	7	78.6%	5	1	1

Judged blind by an LLM. Protocol: both orderings; verdict must survive the swap or it is a tie. Ties count as half a win. Every pair is judged twice with the answers swapped, because pairwise judges have a systematic position preference; a disagreement between the two orderings is recorded as a tie rather than resolved in our favour.

Prompt sets. Held-out domain tasks come from occupations that appear nowhere in training. Out-of-template probes are hand-written and deliberately unlike anything in the dataset — conversational phrasing, under-specified questions, and tasks spanning several skills at once. False-premise prompts assert something untrue and are scored on whether the model corrects it or goes along with it.

On the choice of baseline — read this before comparing

The challenge scores each entry relative to the checkpoint it was fine-tuned from. We chose the base checkpoint (Qwen/Qwen2.5-1.5B), not the Instruct sibling. A base model has no instruction-following behaviour at all, so the measured improvement is large partly because the starting point is low.

That is a real effect and we are not hiding it, which is why the table above also reports the same adapter against Qwen2.5-1.5B-Instruct — a genuine instruction-following model that was never the training baseline. Read that row as the honest measure of what the domain data added.

Training


Method	QLoRA, 4-bit NF4, r=32, alpha=64, dropout=0.05
Loss	completion-only (prompt masked with -100)
Domain rows	3,759 (from `miscusi/adaption-hr-advisory-onet`)
General rows	2,024 (35% of mix, `databricks/databricks-dolly-15k`)
Epochs / LR	3 / 0.0002
Max sequence	1024
Final train loss	0.7685
Runtime	50 min on a single A10G

Loss is masked over the prompt. The model is graded on what it produces, so that is the only region the loss sees. Training on the full sequence teaches the instruction template — precisely the overfitting an unseen-task metric punishes.

A third of the mix is general instruction data. Without it, a base model trained only on domain rows becomes a domain-template completer and collapses on any prompt phrased differently.

Usage

Released as a LoRA adapter (r=32, ~147 MB) under adapter/. Load it on top of the base checkpoint with PEFT — this is the artifact that was trained, and keeping it separate means you can also apply it to a quantised base.

Prompt format is ChatML. Generation terminates on <|endoftext|>, not <|im_end|> — set it or the model will not stop (see the note below).

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen2.5-1.5B"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="float16", device_map="auto")
model = PeftModel.from_pretrained(model, "miscusi/adaption-hr-advisor-qwen2.5-1.5b", subfolder="adapter").eval()

SYSTEM = "You are an experienced HR business partner. Answer practically and concisely, ground advice in what the role actually involves, and say plainly when a common practice is a bad idea."
prompt = (
    f"<|im_start|>system\n{SYSTEM}<|im_end|>\n"
    f"<|im_start|>user\nWhat should we measure in a store manager's performance review?<|im_end|>\n"
    f"<|im_start|>assistant\n"
)
enc = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(
    **enc,
    max_new_tokens=400,
    do_sample=False,
    eos_token_id=tok.eos_token_id,   # <|endoftext|> — required
)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Why the stop token is `<|endoftext|>` and not `<|im_end|>`

<|im_end|> exists in the Qwen2.5 base vocabulary but base pretraining never emits it, so its lm_head row is still at initialisation — measured norm 0.4142, identical to <|im_start|>, against 1.1506 for the native <|endoftext|>. LoRA here adapts attention and MLP only, never lm_head or the embeddings, so an untrained output row is unreachable no matter how much data you show it.

An earlier revision of this model was trained to terminate on <|im_end|>. It learned the domain content correctly and then measured P(<|im_end|>) = 0.000000 at the end of a finished answer: it ran to the token cap on every prompt and degenerated past the point where the answer had clearly ended. If you fine-tune a Qwen2.5 base checkpoint with LoRA, terminate on the model's own EOS.

Mirrors and companion artifacts

The challenge requires the dataset and the weights on Hugging Face and Kaggle. All four artifacts for this track, plus the public demo:

Artifact	Link
Dataset (HF)	https://huggingface.co/datasets/miscusi/adaption-hr-advisory-onet
Dataset (Kaggle)	https://www.kaggle.com/datasets/usiadianimuwa/adaption-hr-advisory-onet
Weights (HF)	https://huggingface.co/miscusi/adaption-hr-advisor-qwen2.5-1.5b
Weights (Kaggle)	https://www.kaggle.com/datasets/usiadianimuwa/adaption-hr-advisor-qwen25-15b
Training dataset (HF)	https://huggingface.co/datasets/miscusi/adaption-hr-advisory-onet
Demo — every eval prompt and all answers, including our losses	https://miscusi-adaption-autoscientist-demo.static.hf.space (Space)

Why trust these numbers

Judged blind, in both orderings; a verdict that does not survive swapping the answers counts as a tie, never a win.
The demo publishes every evaluation prompt and all models' answers, including our losses.
Every numeric claim in the training data was independently re-derived from source by the verify.py shipped with the dataset.

Citation

bibtex

@misc{adaption_hr_advisor_qwen2.5_1.5b_2026,
  author = {Adia-Nimuwa, Usi},
  title  = {adaption-hr-advisor-qwen2.5-1.5b: AutoScientist Challenge Part 2, HR track},
  year   = {2026},
  url    = {https://huggingface.co/miscusi/adaption-hr-advisor-qwen2.5-1.5b}
}

Limitations

1.5B parameters. It is a competent domain assistant, not an expert system, and it will state things confidently that are wrong.
Advice reflects O*NET's US occupational data and US HR practice. Employment law differs by jurisdiction and none of this is legal advice.
English only.
Trained on template-generated data mixed with Dolly; style is consistent but narrow.

Licence

Apache 2.0, following the base model. Training data licensing is documented on the dataset card.

miscusi/adaption-hr-advisor-qwen2.5-1.5b

HR Advisory Model (O*NET-grounded)