Indonesian candidates and parties file campaign-finance reports — LADK,
LPSDK, LPPDK — with the KPU (Komisi Pemilihan Umum). Turning those
filings into auditable data means reading a donor table and emitting one record
per donation, without inventing any.
This is a LoRA adapter for
unsloth/DeepSeek-R1-Distill-Llama-8B
(83.9 M trainable parameters) that does that. It is inert on its own — it needs
the base model.
It pairs with
cakradana-v1-ocr-deepseek-3b-adapter,
which produces the text this model reads. Both were built for
Cakradana, an AI system for transparency in
Indonesian election financing, which placed 3rd in the Open Sector at the
TrackAML 2.0 hackathon run by PPATK, Indonesia's Financial Transaction
Reports and Analysis Center.
📊 Evaluation
Compare only at equal token budget. The base model spends its output budget
reasoning: at 1,024 tokens, 44 of 100 baseline outputs never closed </think>
and so never reached any JSON. Scoring those as failures credits this adapter
for R1-Distill's verbosity rather than for extraction.
The middle column re-runs 30 of the identical chunks at 3,072 tokens (the
eval seeds its selection, so those rows are a strict subset). All three columns
below are those same 30 chunks.
metric
base @1024
base @3072
tuned @1024
parse rate
0.333
0.733
1.000
fabrication rate
0.444
0.643
0.000
span validity
0.867
0.935
0.918
F1 (strict)
0.050
0.198
0.418
recall
0.025
0.136
0.322
cut off mid-reasoning
13
6
0
Fabrication is the result
89% of real filing chunks carry no donation at all — covers, decision
letters, expenditure tables. A model that invents records on those is worse
than useless no matter how it scores elsewhere.
The base model fabricates on roughly two thirds of the empty chunks it manages
to answer, and giving it three times the budget does not help (0.444 →
0.643; more budget means more outputs parse, which reveals more fabrication).
The tuned model fabricates on none — 0 of 60 hard-null chunks in the full
run, while still emitting 218 records on the chunks that do carry them. It
learned to discriminate, not to refuse.
Span validity is not an improvement
At equal budget the base model scores 0.935 against this adapter's 0.918.
The apparent 0.792 → 0.931 gain in the naive 1,024-token comparison is a budget
artefact, and it is not quoted here as a win.
Secondary: the full 100-chunk run
Both columns at 1,024 tokens, so the parse-rate and recall gaps are inflated by
the budget effect above. Reported for completeness, not as the headline.
metric
base
tuned
parse rate
0.370
0.990
fabrication rate
0.636
0.000
F1 (strict)
0.030
0.578
F1 (amount-only)
0.039
0.745
The matcher is not doing the work: folding titles, legal forms and punctuation
moves strict F1 by 0.007. The gap to amount-only matching is real signal —
where this model errs, it more often has the donation right and the donor name
wrong than the reverse.
The published figures were measured against the private pre-redaction corpus.
The script reads the public release — a redacted subset, 6,174 pairs
against 8,016 — so it selects different rows and will not reproduce them to the
digit. It reproduces the comparison.
🚀 Usage
The prompt is not a suggestion. This adapter was trained under
vocabulary.build_system_prompt() / build_user_prompt(), shipped in this
repo. Substituting your own instructions serves the model something it never
saw during training.
bash
1pip install"transformers==4.57.3" peft unsloth
2pip uninstall -y torchao # peft calls is_torchao_available(), which raises3# rather than returning False on some builds
python scripts/extract.py filing.txt
python
1from scripts.extract import load, extract, check_spans
23model, tokenizer = load()# base + this adapter4records, raw = extract(model, tokenizer, document_text)5bad = check_spans(records, document_text)# every *_span must be quotable
add_generation_prompt=True ends the prompt at <｜Assistant｜><think>\n, and
the model continues </think>\n[...]. Training used exactly that state, so the
standard chat-template call is the correct one — no prompt surgery, and no
reasoning trace to pay for. The <think> block is deliberately empty: this task
teaches an output format, and the span rule is what guards correctness.
🗂️ Output schema
A JSON array of objects, each carrying only the fields the document supports:
json
1[{"sender":"Hj. Nessy Ariyani","sender_type":"individual",2"amount":750000,"date":"2015-10-15",3"sender_span":"Hj. Nessy Ariyani","amount_span":"750.000",4"date_span":"15 OKTOBER 2015"}]
Every field carries a *_span, and a span is the raw quoted substring — not
an offset. The value is normalised (750000); the span is what is printed on
the page ("750.000"). This is the contract that makes an extraction checkable
rather than merely plausible: check_spans() in scripts/extract.py verifies
each one against the source, and a span that is not in the document is a
fabricated quote however right the value looks.
An empty array is a valid and common answer. It is the correct answer for most
filing pages.
adamw_8bit, lr 1e-4 linear, warmup 0.03, weight decay 0.01
precision
bf16, max_seq_length 4096
seed
20260819
hardware
1× A100-SXM4-40GB
runtime
1,114.6 s (18 m 35 s)
final train loss
0.0679
Data. 8,016 pairs → 7,984 after deduplicating chunk placements → 343
record-bearing and 6,179 hard-null in train. Hard nulls are downsampled to
3:1 against record-bearing chunks (the corpus ships at 17:1), giving 1,372
rows, then 1,349 after dropping 23 over the sequence limit.
That 3:1 ratio is what produced the fabrication result. It was not swept; a
different ratio would likely trade fabrication against recall.
Loss falls on the answer only, via train_on_responses_only with the
<｜User｜> / <｜Assistant｜> markers. Note those are U+FF5C FULLWIDTH VERTICAL
LINE, not ASCII | — an ASCII pipe matches nothing and silently supervises
zero tokens. scripts/train_extraction.py decodes the supervised span and
aborts unless </think> appears in it, rather than training for hours on
nothing.
The published dataset is a redacted subset of what this run saw. Training
used the private pre-redaction corpus; the public release
(cakradana-kpu-filings-14k-pages)
withholds pages carrying personal identifiers, leaving 6,174 pairs against the
8,016 used here. It is the corresponding citable dataset.
⚠️ Limitations
Recall collapses on record-dense chunks: 0.608 (1–5 records), 0.652
(6–20), 0.241 (21+). The cause is in the recipe above, not the model. The
max_seq_length=4096 filter dropped 13 record-bearing training rows whose
median was 60 records (max 172), leaving a training set that tops out at
45. Of 40 record-bearing eval outputs only 1 was truncated by the output
cap; 23 emitted fewer records than expected and then closed the array —
142 records never emitted. Raising --max-seq to 8192 is the obvious v2, and
the F1 above is its baseline.
Raise max_new_tokens for dense pages. One eval output was truncated
mid-record at 1,024. scripts/extract.py defaults to 2,048.
eval_clean covers 4 publishing hosts, 166 documents. Nothing here
supports a claim about an unseen publisher.
Both evaluation references are teacher-generated, so absolute scores
describe agreement with a teacher, not ground truth.
19.1% of record placements in the training split are duplicated across
overlapping chunks (touching 40.2% of record-bearing chunks). Oversampling,
not leakage: train and eval share zero documents, zero hosts and zero
identical records.
Indonesian campaign-finance documents specifically. Not a general
information-extraction model.
Output is a reading, not a finding. Nothing this model emits is evidence
of wrongdoing by anyone named in a document it read.
📜 Licence
Built with Llama. Governed by the Llama 3.1 Community
License, inherited through
DeepSeek-R1-Distill-Llama-8B from Llama-3.1-8B-Base. That covers the weights;
it does not release you from the training corpus's terms, which prohibit
attempting to re-identify individuals and use for harassment or profiling. See
LICENSE.
