llm-d-sc-sensitivity
A 384-dimensional sentence embedding model fine-tuned to separate
data sensitivity tiers, for use
as a routing signal by
llm-d-sc.
The model does not emit a class directly. It produces an embedding that is ranked against labelled
anchors (anchors.json, shipped here), so the taxonomy stays as replaceable data rather than a
frozen classification head.
Taxonomy
| Tier | Meaning |
|---|
PUBLIC | General knowledge; nothing organization-specific |
INTERNAL | Organization-specific but not sensitive: runbooks, process, ownership |
CONFIDENTIAL | Business-sensitive: unreleased plans, contracts, compensation, security posture |
REGULATED | Legally regulated data: PII, PHI, PCI, financial records |
NEVER_EGRESS | Credentials and key material that must never leave the boundary |
Intended use
Deciding whether a prompt may be served by an external model endpoint. NEVER_EGRESS and
REGULATED prompts typically must stay on in-cluster models. The classifier emits ranked evidence
only; enforcement is the caller's responsibility.
This is a routing signal, not a security control. It is a similarity ranking with a measured
error rate. Do not use it as the sole barrier preventing secret exfiltration.
Evaluation
Evaluated by llm-d-sc on a held-out set of 75 prompts authored independently of the training
corpus, 15 per tier, of which 25 are deliberate boundary cases (for example "What makes a password
strong, in general terms?" is PUBLIC, not NEVER_EGRESS).
Method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.
| Model | Accuracy | Macro F1 | Boundary cases |
|---|
| llm-d-sc-sensitivity (this model) | 0.8933 | 0.8928 | 0.7600 |
all-MiniLM-L6-v2 (base, same anchors) | 0.7067 | 0.6920 | 0.6400 |
Per tier:
| Tier | Precision | Recall | F1 | Support |
|---|
| PUBLIC | 0.929 | 0.867 | 0.897 | 15 |
| INTERNAL | 0.846 | 0.733 | 0.786 | 15 |
| CONFIDENTIAL | 0.778 | 0.933 | 0.848 | 15 |
| REGULATED | 0.933 | 0.933 | 0.933 | 15 |
| NEVER_EGRESS | 1.000 | 1.000 | 1.000 | 15 |
NEVER_EGRESS separates perfectly: credential-bearing text is lexically and semantically distinctive.
Residual error concentrates on the INTERNAL / CONFIDENTIAL boundary, which is genuinely a matter
of organizational policy rather than of language.
Latency on CPU (single thread, Apple M-series, embed plus rank): p50 8.1 ms, p99 12.9 ms.
These numbers were produced on a homelab and have not been independently reproduced.
Known limitation: training data quality
The synthetic data pipeline that produced this model captured the generator's reasoning traces
rather than the intended generated texts. Training sentences average 175 tokens and read like
"The user wants a NEW example text that belongs to the PUBLIC sensitivity tier..." instead of like
real user prompts.
The model still performs well above the base model, because anchor ranking depends on the geometry
of the embedding space rather than on the training text distribution. But the gap between this model
(0.8933) and the equivalently trained complexity model (0.9750, clean training data) is the likely
cost of that defect. Retraining on corrected data is expected to close it.
Training
Fine-tuned from
sentence-transformers/all-MiniLM-L6-v2 with
BatchAllTripletLoss and
group_by_label batch sampling: 495 synthetic examples, 20 epochs, batch size 16, lr 2e-5.
Pipeline:
https://github.com/cnuland/hello-chris-sr-finetunedLimitations
- English only.
- Trained on synthetic data with the defect described above; no human-labelled validation set exists.
- Anchor quality directly determines accuracy. Replacing
anchors.json changes behaviour without retraining.
- Sensitivity is organization-specific. The shipped anchors encode one reasonable policy, not yours.
License
Apache-2.0.
