underdog-lab-smollm2-360m-lora
A LoRA adapter over SmolLM2-360M-Instruct for structured factor extraction from
football match scenario text.
This adapter was evaluated against a pre-declared gate and did not pass. It is
not the production path. It is published because the negative result is the
useful part — the gate, the frozen test set, and the numbers below are the
artifact. See
Evaluation.
Model details
- Developed by: Osama Moftah (
sammoftah)
- Model type: LoRA adapter (PEFT) for a causal LM
- Base model: HuggingFaceTB/SmolLM2-360M-Instruct
- Language: English
- License: Apache-2.0
- Adapter size: 3.3 MB (
adapter_model.safetensors)
Sources
- Training data: sammoftah/underdog-lab-scenarios
- Project: Underdog Lab — scenario extraction for a World Cup 2026 forecaster
- Write-up: https://moftah.dev/work/underdog-eval-gate
What it does
Given a free-text match scenario, the model emits a structured record of
factors — the things in the text that should move a forecast. Each factor
carries a type, the team it applies to, a severity and certainty in [0, 1], and
the span of evidence it was drawn from. The schema also has explicit slots for
unsupported_claims and ambiguities, so the model is asked to mark what it
could not ground rather than quietly inventing it.
Intended use
Research and replication of the evaluation below. Not intended for production
use — the base model plus a deterministic fallback outperforms it on the
metrics that matter (see below).
Out of scope
Anything requiring calibrated severity or certainty values. The central failure of
this adapter is that it emits near-zero severity and certainty, which a downstream
zero-weight check correctly flags as unusable.
Training
| |
|---|
| Method | LoRA (PEFT 0.19.1) |
Rank r | 16 |
lora_alpha | 32 |
lora_dropout | 0.05 |
| Target modules | q_proj, v_proj |
| Bias | none |
| Task type | CAUSAL_LM |
| Train / validation | 861 / 56 examples |
Evaluation
Benchmarked 2026-06-13 against a frozen 98-example test set
(data/scenarios/test.jsonl), base vs. tuned, both quantised to Q8_0 and served
through the same runtime. Thresholds were declared before the run.
| Metric | Base | Tuned | Δ | Threshold | |
|---|
| Factor micro-F1 | 0.0261 | 0.0270 | +0.0010 | ≥ +0.15 gain | ❌ |
| Team attribution accuracy | 0.0430 | 0.0323 | −0.0108 | no regression | ❌ |
| Fallback rate | 0.0918 | 0.1735 | +0.0817 | ≤ +0.02 | ❌ |
| Schema validity | 1.000 | 1.000 | 0.000 | ≥ 0.99 | ✅ |
| Median latency | 3675.6 ms | 2995.4 ms | −680.2 ms | — | — |
Result: 3 of 4 checks failed. Decision: NO-SHIP.
The adapter got marginally better at classifying factor type and materially
worse at everything else. It emits near-zero severity and certainty, which trips
the zero-weight hallucination check, which is why the fallback rate nearly
doubled. It is faster, and speed was not one of the gates.
The production path remains base model + deterministic fallback.
Why this is published
A fine-tune that fails a gate is only a wasted run if you throw away the gate. The
threshold table, the frozen test set, and the base-vs-tuned comparison are
reusable; the weights are the least interesting output. Publishing the adapter
alongside the report is what makes the negative result checkable by someone else.
Limitations and bias
- Trained on synthetic and semi-synthetic football scenarios; it will not transfer
to other domains.
- Severity and certainty outputs are not calibrated and should not be read as
probabilities.
- Micro-F1 is low in absolute terms for both base and tuned models — this task
is not solved at 360M parameters. The comparison is valid; the ceiling is low.
- Evaluated in English only.
How to use
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base = "HuggingFaceTB/SmolLM2-360M-Instruct"
5model = AutoModelForCausalLM.from_pretrained(base)
6model = PeftModel.from_pretrained(model, "sammoftah/underdog-lab-smollm2-360m-lora")
7tok = AutoTokenizer.from_pretrained(base)
Citation
1@misc{moftah2026underdoglora,
2 author = {Moftah, Osama},
3 title = {underdog-lab-smollm2-360m-lora: a LoRA adapter and the eval gate that blocked it},
4 year = {2026},
5 url = {https://huggingface.co/sammoftah/underdog-lab-smollm2-360m-lora}
6}
Contact
