TF-IDF + logistic-regression cascade for legal-guidance conversations
This repository contains the frozen full-conversation lexical baseline for the
Legal Guidance in the Wild study. It first predicts whether the user is
seeking legal guidance and, for predicted-positive conversations, assigns one
of 14 primary legal topics. The models receive chronological user messages
only; assistant responses are excluded.
This is a research classifier, not a legal-advice system. Its output must not
be treated as a determination of a person's rights, claims, or legal needs.
topic_tfidf_logreg.joblib: fitted 14-way topic model, vectorizers, and topic
names
guidance_metrics.json, topic_metrics.json, and
cascade_val_metrics.json: complete silver-validation metrics
majority_metrics.json: majority-baseline results
validation_predictions.csv: all 290 frozen validation predictions
run_metadata.json: data fingerprints, software versions, and search space
Model design
Each stage concatenates word unigram/bigram TF-IDF features (maximum 150,000)
with character-within-word 3--5-gram TF-IDF features (maximum 200,000), then
fits class-balanced logistic regression. The regularization value was selected
from C = {0.25, 1, 4}; both stages selected C = 1. The guidance threshold
was selected on the silver validation split and is stored in the artifact
(0.51 for this run).
Dataset access follows the sharing settings of the linked dataset repository.
The fingerprints above identify the exact loaded splits even if main later
changes.
Silver-validation results
Stage
Metric
Value
Guidance (N=290)
Macro-F1
0.8653
Guidance (N=290)
Positive-class F1
0.8602
Guidance (N=290)
Accuracy
0.8655
Guidance (N=290)
Balanced accuracy
0.8676
Guidance (N=290)
AUPRC
0.9105
Topic, guidance-positive only (N=134)
Accuracy
0.6567
Topic, guidance-positive only (N=134)
Macro-F1
0.4927
End-to-end 15-way cascade (N=290)
Accuracy
0.7310
End-to-end 15-way cascade (N=290)
Macro-F1
0.4514
At threshold 0.51, the guidance confusion matrix is TN=131, FP=25, FN=14,
TP=120. These are silver-validation results used for development and model
selection; they are not final adjudicated-gold estimates.
Loading the cascade
Install compatible dependencies:
pip install "scikit-learn==1.6.1" joblib
The following shows the artifact structure and cascade logic. texts must use
the same chronological, user-only serialization as training.
python
1import joblib
2from scipy.sparse import hstack
34texts =["My landlord is withholding my deposit. What can I do?"]56guidance = joblib.load("guidance_tfidf_logreg.joblib")7Xg = hstack([8 guidance["word_vectorizer"].transform(texts),9 guidance["char_vectorizer"].transform(texts),10],format="csr")11guidance_probability = guidance["model"].predict_proba(Xg)[:,1]12seeks_guidance = guidance_probability >= guidance["threshold"]1314topic = joblib.load("topic_tfidf_logreg.joblib")15Xt = hstack([16 topic["word_vectorizer"].transform(texts),17 topic["char_vectorizer"].transform(texts),18],format="csr")19topic_id = topic["model"].predict(Xt)20topic_name =[topic["topic_names"][int(i)]for i in topic_id]2122cascade_prediction =[23 name if positive else"NO_GUIDANCE"24for positive, name inzip(seeks_guidance, topic_name)25]
joblib uses pickle-based serialization. Load artifacts only from a repository
and revision you trust.
Limitations
The models were trained on English-language public LLM interaction logs with
silver labels, one data source, and one random seed. Rare topics have very small
validation support, and three topics have zero validation F1 in this run. The
dataset is jurisdiction-agnostic, may contain sensitive material, and is not
representative of all people who seek legal help. Threshold calibration and
performance may shift in other platforms, jurisdictions, time periods, or
deployment populations. Human review is required for consequential use.
Please cite the accompanying Legal Guidance in the Wild: How Users Seek Legal
Help in Real-World LLM Conversations manuscript when it becomes available.
