Model card — kid-extract
LoRA adapter for HuggingFaceTB/SmolLM2-135M-Instruct that extracts a fixed JSON schema from
European fund disclosure documents (PRIIPs KID and UCITS KIID).
Intended use
Converting the text of a fund disclosure document into validated JSON: risk indicator,
charges, recommended holding period, investment objective, performance scenarios and fund
identifiers. Built for batch processing where cost and latency per document matter.
Runs on CPU. No GPU is required for training or inference.
Out of scope
- Investment advice, suitability assessment, or any decision affecting a retail investor
- Documents outside the PRIIPs and UCITS disclosure families
- Languages other than English, German, French and Dutch
- Any use where an unverified extraction feeds a regulatory filing or a client-facing figure
Output is machine-readable, not authoritative. Validate before relying on it.
Training data
Synthetic documents generated by the kid-extract repository, grounded optionally in real
fund attributes from the Morningstar European funds dataset. See the dataset card for how the
corpus is built and what it does not capture. No real disclosure document was used for
training, and nothing proprietary or confidential was involved.
Training procedure
| |
|---|
| Base model | HuggingFaceTB/SmolLM2-135M-Instruct |
| Method | LoRA (PEFT) via TRL SFTTrainer |
| Trainable parameters | 4,884,480 of 139,399,488 (3.5%) |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Target modules | q_proj k_proj v_proj o_proj gate_proj up_proj down_proj |
| Max sequence length | 1408 |
| Learning rate / schedule | 0.0002 / cosine |
| Batch / accumulation | 1 / 4 |
| Training examples / epochs | 1500 / 1 |
| Optimiser steps | 375 |
| Final train / eval loss | 0.0478 / 0.0057 |
| Wall clock | 5.7 hours on 4 CPU threads |
| Hardware | Intel i5-8365U, 4 cores, CPU only |
Loss is computed on the JSON completion alone. The prompt contains a field specification that
is byte-identical in every example, so training the model to reproduce it would spend capacity
on nothing.
An earlier run at 38 optimiser steps was measurably undertrained: it produced unbalanced JSON
and runaway generation, and raising the token cap from 400 to 640 did not help. Gradient
accumulation was halved from 8 to 4 to buy ten times the weight updates for the same wall
clock. Anyone shortening this run should expect schema validity to degrade first.
Evaluation
Scored on test_unseen_layout, which combines four layouts held out of training with the
reserved label vocabulary — wordings for every field that appear in no training document.
This is deliberately the harder of the two test splits.
Metrics: per-field precision, recall and F1 over 23 fields; null accuracy; schema validity;
hallucination rate measured against the source text; exact match; median latency per document.
Reference point
A hand-written multilingual regular expression extractor, given the training vocabulary only:
| Split | Micro F1 | Macro F1 | Exact match |
|---|
test_seen | 0.978 | 0.976 | 0.633 |
test_unseen_layout | 0.645 | 0.516 | 0.000 |
The collapse on unseen wordings is the gap this adapter is trained to close.
Results
All systems scored on the same 50 documents by identical code.
Unseen layouts and unseen label wordings:
| System | Micro F1 | Macro F1 | Exact | Schema valid | Hallucination | Latency |
|---|
| fine-tuned 135M | 0.858 | 0.830 | 0.02 | 0.88 | 0.005 | 21.3 s |
| rules | 0.643 | 0.514 | 0.00 | 1.00 | 0.000 | 0.001 s |
| few-shot 135M | 0.169 | 0.155 | 0.00 | 0.50 | 0.491 | 40.0 s |
| zero-shot 135M | 0.000 | 0.000 | 0.00 | 0.00 | 0.000 | 28.1 s |
Known layouts and wordings:
| System | Micro F1 | Macro F1 | Exact | Schema valid | Hallucination | Latency |
|---|
| fine-tuned 135M | 0.988 | 0.988 | 0.82 | 1.00 | 0.006 | 21.1 s |
| rules | 0.983 | 0.982 | 0.72 | 1.00 | 0.000 | 0.001 s |
The same base model goes from producing no valid JSON at all to 0.858 micro F1 on
documents whose layout and vocabulary it has never seen. Few-shot prompting reaches only
0.50 schema validity and invents 49% of the values it emits.
Rules remain better on fields recoverable by shape rather than by name — ISIN, fund name, the
scenario table, the risk scale — and run 20,000 times faster. The model wins on every
label-dependent field, where an unfamiliar wording sends a regular expression to zero. A hybrid
would beat both.
Still weak: transaction_costs_pct at 0.078 and domicile at 0.350 on unseen wordings, schema
validity dropping to 0.88, and exact match over all 23 fields at once of 0.02. Per-field
accuracy is not whole-record accuracy; validate fields individually rather than trusting a record.
Limitations
- Trained on generated documents. Performance on real provider PDFs is unmeasured until the
hand-labelled real test set exists, and should be assumed lower.
- A 135M model has little capacity for reasoning about a value that is implied rather than
stated. Fields requiring inference are out of reach by design; the prompt instructs the
model to copy, not calculate.
- Long documents are truncated at 1408 tokens.
- Text must already be extracted from the PDF. No OCR or layout analysis is included.
Licence
MIT. Base model is Apache 2.0.
