A weakly-supervised multi-label classifier that reads a free-text chest-CT report and predicts a 137-leaf chest-imaging taxonomy, with a ternary status per label (negative / uncertain / positive).

It also provides a CheXbert / SRR-BERT-style report-comparison F1: label a list of ground-truth reports and a list of generated/predicted reports, then score them against each other (micro / macro / weighted F1) — useful for evaluating radiology report generation.

Base architecture: Qwen/Qwen3-Embedding-0.6B (Apache-2.0)
Adaptation: LoRA (r=16, α=32) merged into the weights + last-token (EOS) pooling + L2-norm + a linear ternary head (1024 → 137 × 3)
Self-contained: the full model (encoder + head) ships in model.safetensors. Loading does not download Qwen3-Embedding weights — the architecture is rebuilt from the bundled config and our weights are loaded in. Tokenizer is bundled too.
Params: ~596M · weights in float32
Training labels: chest2vec/chest2vec_labels (revised CT-RATE, 137-leaf taxonomy)

Label space

The model head predicts 137 leaf labels. They roll up through the chest-imaging hierarchy into 38 upper/container groups and 10 anatomy sections (the label_hierarchy in config.json), so predictions and report-comparison F1 can be reported at leaf, upper, or anatomy granularity.

The model outputs all 137 leaves. In the training data, 136 of them have at least one positive example; the single exception is IVC filter (kept for taxonomy completeness, but it had no positives, so the model effectively never predicts it).
The exact label list is in config.json (labels). Full definitions and per-split counts are in the chest2vec/chest2vec_labels dataset's LABEL_HIERARCHY.md.

This model was trained and evaluated on the chest2vec/chest2vec_labels dataset (revised CT-RATE, 137-leaf taxonomy).

Ternary head — softmax(logits, dim=-1) over class indices [0, 1, 2]:

class index	meaning	value
0	negative	0
1	uncertain	-1
2	positive	1

A label is reported positive when P(class=2) ≥ threshold (default 0.5).

Usage

python

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("chest2vec/chest2vec_labeler", trust_remote_code=True).eval()
tok   = AutoTokenizer.from_pretrained("chest2vec/chest2vec_labeler", trust_remote_code=True)

reports = ["Bibasilar atelectasis with small bilateral pleural effusions. Cardiomegaly. Coronary artery calcification."]

# 1) human-readable positive labels per report
print(model.label_reports(reports, tokenizer=tok))
# [{'Subsegmental / linear atelectasis': 'positive', 'Pleural effusion': 'positive',
#   'Cardiomegaly': 'positive', 'Coronary artery calcification': 'positive'}]

# 2) full prediction matrices
out = model.predict(reports, tokenizer=tok, threshold=0.5, return_ternary=True)
out["labels"]      # list of 137 label names
out["proba"]       # [N, 137] P(positive)
out["positive"]    # [N, 137] in {0,1}
out["ternary"]     # [N, 137] in {-1,0,1}

CheXbert / SRR-BERT-style report comparison

Label both ground-truth and predicted reports, then compute label-level F1 (GT-labels treated as truth):

python

res = model.score_reports(gt_reports, pred_reports, tokenizer=tok)   # equal-length lists
# scores are reported at three hierarchy levels:
for level in ("leaf", "upper", "anatomy"):
    b = res[level]
    print(level, b["n_labels"], b["micro"]["f1"], b["macro"]["f1"], b["weighted"]["f1"])
print(res["leaf"]["per_label"]["Pleural effusion"])   # {'precision':..,'recall':..,'f1':..,'support_gt':..}

# or one-liner that loads the model for you:
from modeling_chest2vec_labeler import report_f1
report_f1(gt_reports, pred_reports, tokenizer=tok)

Each level (leaf = 137 labels, upper = 38 container groups, anatomy = 10 sections) returns micro / macro / weighted precision/recall/F1 plus per_label. Upper/anatomy scores are the max-over-children roll-up of the leaf predictions (model.aggregate_hierarchy(...)). Coarser levels are easier to match, so upper/anatomy F1 are typically higher than leaf.

Per-label best F1 (threshold tuning)

The default decision threshold is a single global value, but the F1-optimal threshold differs per label. To get the best achievable F1 per label (and the threshold that achieves it) against a ground-truth label set:

python

# gt: a DataFrame with the 137 label columns (ternary; positive == 1), or a binary array
res = model.per_label_best_f1(reports, gt, tokenizer=tok, level="leaf", min_pos=30)
res["macro_best_f1_min_pos"]                 # macro best-F1 over labels with >= min_pos positives
res["per_label"]["Pleural effusion"]         # {'best_f1':.., 'best_threshold':.., 'n_pos':..}

Per-label threshold tuning lifts macro-F1 by ~4–6 points over the fixed-0.5 threshold (see below).

Inputs & conventions

Input is the findings text (the model was trained on CT-RATE findings + their refined section-structured form). Reports are formatted internally as Instruct: Given the following chest CT report, extract the presence/absence of entities\nQuery: <report>, truncated to 512 tokens, with an EOS token appended and left-padding.
For best fidelity, run in float32 (default). bf16 is fine for throughput with negligible drift.

Evaluation

Reported as macro-averages over per-label leaf metrics, with 95% bootstrap percentile confidence intervals. Compare only within a row.

AUROC — macro one-versus-rest AUROC on the positive-class probability. Threshold-free.
Three-class F1 — macro F1 from the ternary head (negative / uncertain / positive). Uses argmax; no threshold involved.
Binary F1 — macro F1 with uncertain mapped to negative. Per-label decision thresholds are selected on the CT-RATE tuning partition (not on the reported eval set).

Two label subsets are reported per eval set:

≥30 positives — leaves with at least 30 positive test instances (headline).
≥1 positive — all evaluated leaves (of the 137, 129 had ≥1 positive in CT-RATE and 120 in SNUH).

Label set	AUROC	Three-class F1	Binary F1
CT-RATE (in distribution)
53 (≥30 pos)	0.989 (0.988, 0.991)	0.686 (0.667, 0.692)	0.888 (0.879, 0.896)
129 (≥1 pos)	0.988 (0.985, 0.991)	0.610 (0.592, 0.612)	0.769 (0.731, 0.777)
SNUH (external)
29 (≥30 pos)	0.971 (0.967, 0.975)	0.590 (0.579, 0.598)	0.766 (0.749, 0.780)
120 (≥1 pos)	0.972 (0.968, 0.975)	0.541 (0.523, 0.543)	0.652 (0.601, 0.660)

Leaf macro-AUROC barely moves in distribution → external (0.989 → 0.971): label ranking transfers to unseen data; the residual F1 gap reflects labeling-convention drift, not a domain failure. A radiologist spot-checked 966 reports of the public test labels (857 fully accepted / 60 imperfect-but-acceptable / 49 failed; see the dataset card).

Caveats

Weakly supervised — trained on LLM-generated labels (not radiologist ground truth) derived from report text, not images. Not a medical device; not for clinical use.
IVC filter is in the taxonomy for completeness but had no training positives.
score_reports measures label agreement between two reports as judged by this labeler; like CheXbert-F1 it inherits the labeler's own error modes.

License & attribution

Released under CC-BY-NC-SA-4.0. Built on Qwen/Qwen3-Embedding-0.6B (Apache-2.0) and trained using labels derived from CT-RATE (CC-BY-NC-SA-4.0). If you use this model, cite the CT-RATE paper (arXiv:2403.17834) and acknowledge Qwen3-Embedding. See the dataset card for the full citation.