Lilly · Fashion Muse — doc-level review scorer. 3-class doc-level review sentiment (negative / neutral / positive) with a selective-prediction gate: predictions whose max softmax probability is below tau = 0.70 are abstained (UNSURE / MIXED) instead of forced.

Measured operating point (held-out, n = 4,529)

tau	coverage	committed acc	committed macroF1	abstains
.70	86.2%	91.2%	.715	13.8%
.80	77.8%	95.1%	.690	22.2%

Uncalibrated full-test baseline (same weights, no gate): acc .855 / macroF1 .695.

Honest limitations

Weak-label provenance: training labels are derived from review-star ratings (3 -> neutral, <=2 -> negative, >=4 -> positive), not human annotation of sentiment.
Annotation ceiling: intra-annotator kappa on this dataset is ~0.15 (collapsed agreement 50%) — absolute accuracy numbers must be read against that ceiling. The gate, not raw accuracy, is the product.
Abstained bucket is dominated by genuinely ambiguous mixed reviews (e.g. "beautiful pattern but runs extremely small").
Committed bucket skews positive as tau rises.

Artifacts & loading

Weights are fp16 shards (fp16_shard_000.pt, fp16_shard_001.pt) that must be merged before load_state_dict (strict):

python

import torch, glob
from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer

state = {}
for p in sorted(glob.glob("fp16_shard_*.pt")):
    state.update(torch.load(p, map_location="cpu"))
cfg = AutoConfig.from_pretrained(".")
model = AutoModelForSequenceClassification.from_config(cfg)
model.load_state_dict(state)          # strict
tok = AutoTokenizer.from_pretrained(".")
model.float().eval()                  # fp32 compute

Verified round-trip: 100.0% prediction agreement with the pre-shard fp32 model. int8 dynamic quantization was tried and rejected (does not round-trip).

Max sequence length at train time: 128. Labels: 0=negative, 1=neutral, 2=positive.

sandeep1103/lilly-fashion-muse-doc-gated

DistilBERT Sentiment (doc-level, confidence-gated)

Measured operating point (held-out, n = 4,529)

Honest limitations

Artifacts & loading

sandeep1103/
lilly-fashion-muse-doc-gated