This lightweight baseline detects whether legal guidance is already present in a cumulative user-turn prefix. It is a research baseline, not a legal-advice system, and must not be used to decide whether a person has a valid claim.

Method

The model combines word (1,2)-gram and char_wb (3,5)-gram TF-IDF features with logistic regression. Prefixes before first_guidance_user_turn_id are negative; the onset prefix and all subsequent prefixes are positive. Each prefix is weighted by the inverse of the conversation's number of user turns, so every conversation has total training weight 1.0.

Both the regularization parameter and probability threshold were selected on the silver validation split by conversation-weighted prefix macro-F1, with weighted positive-class F1 as the tie-breaker.

Data and configuration

Dataset: AmirMohseni/WildChat-Legal-Classification-V3-Hierarchical
Requested revision: main (latest at execution)
Resolved revision: 489b87c18e571e8c7e9a3b1c2463b79e50cef120
Training: 1,632 conversations / 4,865 prefixes
Validation: 290 conversations / 912 prefixes
Selected C: 0.25
Selected threshold: 0.42
Seed: 42
W&B run: https://wandb.ai/rl-research-team/legal-guidance-prefix-v3/runs/fru3wk9m

First-guidance-turn annotations are silver and were not independently human-validated for this run. Onset metrics must not be described as gold.

Validation results

Evaluation	Macro-F1	Positive F1	AUPRC	Accuracy	Balanced accuracy
All prefixes, conversation-weighted	0.8600	0.8387	0.8850	0.8632	0.8664
All prefixes, unweighted	0.8269	0.8079	0.8452	0.8289	0.8354
Final/full prefixes	0.8787	0.8699	0.9199	0.8793	0.8789
Majority reference, weighted prefixes	0.3740	0.0000	0.4026	0.5974	0.5000

Silver onset analysis

Measure	Value
Negative-conversation false-alarm rate	0.1474
Pre-onset false-alarm rate	0.1269
Missed-guidance rate	0.0970
Exact onset accuracy	0.7164
Within-one-user-turn accuracy	0.8209
Mean absolute turn error among detected positives	0.4215

Repository artifacts

tfidf_prefix_pipeline.joblib: fitted feature union, classifier, threshold, and provenance
prefix_validation_metrics.json: complete metrics, candidate results, and selected threshold curve
prefix_validation_predictions.csv: per-prefix probabilities and predictions
onset_validation_predictions.csv: conversation-level onset predictions
run_metadata.json: data, feature, software, and split provenance

Limitations

The labels and onset locations used here are silver. This is a sparse lexical baseline and may be brittle to paraphrases, spelling changes, domain shift, and jurisdiction-specific terminology. It has not yet been evaluated on the frozen human gold set. Gold onset evaluation requires manual review of first_guidance_user_turn_id.

Loading

Only load pickle/joblib files from sources you trust.

python

import joblib

artifact = joblib.load("tfidf_prefix_pipeline.joblib")
X = artifact["features"].transform(["User: example text"])
probability = artifact["classifier"].predict_proba(X)[:, 1]
prediction = (probability >= artifact["guidance_threshold"]).astype(int)

AmirMohseni/tfidf-logreg-v3-legal-guidance-prefix-seed42

TF-IDF + logistic regression for prefix-aware legal-guidance detection