episodic-ingestion-compiler / ModernBERT field-event ranker (mixed-mode v2, H4-320)
Fine-tune of answerdotai/ModernBERT-base for the grouped multi-positive
softmax ranker task, trained on v2 semantic-reasoning labels for
320 optimizer steps (vs 160 in the initial v2 checkpoint). The extra
training let the model fully converge on both legacy and semantic fields.
Versus v2 at 160 steps
| Metric | v2-H4 (160) | v2-H4 (320) | Δ |
|---|
| overall MRR | 0.526 | 0.583 | +0.057 |
| top-1 | 0.296 | 0.375 | +0.079 |
| last loss | 0.0854 | 0.0000 (converged) | |
Training details
- Train rows: 1217 (mixed-mode v2)
- Train groups: 5882
- Eval rows: 220 (21986 candidate pairs)
- Steps: 320 optimizer steps × accum=8 = 2560 forwards
- Optimizer: AdamW lr 5.7e-5, warmup 30
- Precision: bf16 autocast + gradient checkpointing
- Peak VRAM: 4.08 GiB
- Hardware: RTX 5090 Laptop (Blackwell SM 12.0, 24 GiB)
Eval metrics
| metric | value |
|---|
| overall MRR | 0.583 |
| top-1 recall | 0.375 |
| top-2 recall | 0.580 |
| top-3 recall | 0.737 |
| top-5 recall | 0.909 |
| mean expected rank | 2.67 |
Per-field MRR (top fields in eval):
| field | n | MRR | top-1 | random | ratio |
|---|
attempted_actions | 846 | 0.480 | 0.24 | 0.253 | 1.90x |
observed_outcomes | 845 | 0.491 | 0.24 | 0.253 | 1.94x |
initiating_command | 199 | 0.966 | 0.94 | 0.275 | 3.51x |
outcome_of_latest_attempt | 181 | 0.966 | 0.94 | 0.276 | 3.50x |
action_causality | 141 | 0.561 | 0.30 | 0.254 | 2.21x |
attempt_outcome_pairs | 53 | 0.616 | 0.34 | 0.265 | 2.33x |
failed_attempts | 36 | 0.705 | 0.53 | 0.256 | 2.75x |
next_actions | 32 | 0.755 | 0.56 | 0.296 | 2.55x |
recent_error | 29 | 0.826 | 0.72 | 0.256 | 3.23x |
customer_identity | 11 | 0.811 | 0.73 | - | - |
transaction_reference | 5 | 0.633 | 0.40 | - | - |
product_name | 4 | 1.000 | 1.00 | - | - |
discarded_options | 2 | 0.375 | 0.00 | - | - |
invalidation_hints | 2 | 0.750 | 0.50 | - | - |
non_promotable_context | 2 | 0.417 | 0.00 | - | - |
payment_or_warranty_detail | 2 | 0.333 | 0.00 | - | - |
assistant_claims_to_verify | 1 | 0.500 | 0.00 | - | - |
explicit_decisions | 1 | 1.000 | 1.00 | - | - |
resolved_context | 1 | 0.333 | 0.00 | - | - |
touched_files | 1 | 0.200 | 0.00 | - | - |
unsupported_hypotheses | 1 | 1.000 | 1.00 | - | - |
Lineage
| checkpoint | labels | steps | MRR | top-1 | notes |
|---|
| V2 (commit 87cb089) | conversational | 160 | 0.678 | 0.440 | 8-row adversarial eval only |
| mixed-mode stage-4 | v1 | 160 | 0.323 | 0.130 | pre-perf stack |
| mixed-mode perf-H4 | v1 | 160 | 0.506 | 0.260 | perf-H4 stack but v1 (role-tautological) labels |
| mixed-mode v2-H4 | v2 | 160 | 0.526 | 0.296 | first v2 train — non-converged |
| mixed-mode v2-H4-320 (this) | v2 | 320 | 0.583 | 0.375 | converged — strongest checkpoint |
Semantic-reasoning fields show clear learning
The fields designed to NOT be role-tautological reached high MRR through
training, confirming the model is actually reasoning beyond role:
initiating_command: MRR 0.966 (3.51x random) — learns trace boundariesoutcome_of_latest_attempt: MRR 0.966 (3.50x random) — matches call_id chainsrecent_error: MRR 0.826 (3.23x random) — latest-failure reasoningaction_causality: MRR 0.561 (2.21x random) — content grounding across turns
Legacy fields (attempted_actions, observed_outcomes) sit near 0.48,
close to random for multi-positive softmax. They're role-tautological
on 97% of rows, so the model can hit this with role-inference alone.
Intended use
Ranker head of a multi-stage episodic-ingestion compiler. Input is a
JSON-serialized (requested field, candidate event) record. Output is a
scalar logit that, after grouped softmax over siblings in the same trace,
estimates the probability that the candidate event supports the field.
See docs/ranker-hypothesis-log-2026-05-08.md in the repo for the full
experimental ladder (H1-H3b-span falsified, label-pivot kept, H3b-replay-v2
falsified, v2-H4-320 kept).
