A QLoRA adapter that turns Qwen2.5-7B-Instruct into a data-reliability
incident writer. Given a structured finding from a metadata catalog, it
produces a JSON incident report and judges the severity.
Built for DataPhylax, a
three-agent system that watches a DataHub catalog for the failures nobody
notices.
1{2"title":"fintech_core.customer_features lineage broken: upstream clean_transactions deleted",3"severity":"high",4"severity_rationale":"Tier-1 asset with no owner and a deleted upstream. Not critical because no regulated data is exposed; not medium because two production models depend on it and no workaround exists.",5"summary":"...",6"blast_radius":{"downstream_count":2,"affected_assets":[...],"business_impact":"..."},7"recommended_action":{8"steps":["...","...","..."],9"fix_type":"pipeline_repair",10"estimated_effort":"medium",11"requires_human_approval":true,12"owner_action_required":true13}14}
Six event types are supported: freshness_violation, ownership_gap,
pii_regression, silent_duplicate, lineage_break, schema_drift.
Evaluation
40 held-out scenarios the model had never seen, generated with a fresh seed.
Metric
Result
Valid JSON
40/40 (100%)
Schema compliant
40/40 (100%)
Hallucinated identifiers
0/40 (0%)
Severity — exact match vs teacher
74%
Severity — within one level
97%
Throughput (Q4_K_M, A10G)
~76 tok/s
The severity number needs context. The teacher model, run twice on the same
40 inputs at temperature 0, agreed with itself only 82% of the time —
severity is a genuine judgement call, not a fact. So 74% is roughly 90% of
the achievable ceiling, not 74% of a perfect score.
The hallucination check matters more than it looks. Every identifier in the
output is compared against the identifiers present in the input. Zero
inventions means a downstream agent can safely generate code from these
reports without referencing a column that does not exist.
Data: 499 synthetic (finding → report) pairs, distilled from
DeepSeek V4 Pro
Loss masking: only the report contributes to the loss, not the prompt
Best checkpoint: epoch 2 (eval loss 0.5003; epoch 3 rose to 0.5137)
Hardware: single A10G, ~28 minutes
Training data was generated across five industry domains (e-commerce,
fintech, healthcare, logistics, adtech), four severity levels, and varying
lineage depth, so the model learns to read the evidence rather than pattern
match on the event type. Severity was never supplied as an input — the
teacher judged it from the details, and the student learned that judgement.
System prompt used at inference — deliberately short, because the severity
rubric lives in the fine-tune rather than the prompt:
You are a Data Reliability Engineer for a DataHub catalog. Given an incident
event as JSON, return an incident report as a single JSON object. No
markdown, no commentary.
Limitations
Metadata only. It never sees data, so it cannot verify that a column
flagged as PII actually contains PII — only that the metadata says so.
Severity is calibrated to one rubric. An organisation with different
risk tolerances would want to re-tune, not just re-prompt.
English only, and the training data uses Western-style column naming.
Catalogs with different conventions are handled by the detector's
configurable pattern list rather than by the model.
Trained on synthetic findings. Real catalogs are messier than the
scenario generator, though the model was evaluated against DataHub's own
sample catalog and handled unfamiliar naming without difficulty.
