Views
No views yet
intfloat/multilingual-e5-base
(a multilingual XLM-RoBERTa bidirectional encoder). It detects 9 PII categories as
character-offset spans and is trained for multi-domain Korean coverage
(conversational, news, and a range of document domains).| Category | Description | Example |
|---|---|---|
private_person | Personal name (Korean / Western / handles) | 김민수, John Smith |
private_address | Physical / postal address | 서울특별시 강남구 테헤란로 123 |
private_phone | Phone number | 010-1234-5678 |
private_email | Email address | minsu@example.com |
private_date | Birthday / personally-identifying date | 1985년 3월 12일 |
private_url | Personal URL | github.com/minsu |
account_number | Bank, card, RRN, passport, etc. | 110-234-567890 |
personal_handle | Username / handle | rainbow879612 |
ip_address | IP address | 192.168.1.5 |
extract_pii below).| eval set | what it measures | Overall F1 |
|---|---|---|
| KDPII test (2,252) | conversational Korean (in-domain) | 0.943 |
| Held-out document domains (insurance, government) | unseen domains | 0.995 |
KLUE-NER person | real Korean news text | 0.866 (recall 0.92) |
| label | F1 | label | F1 | |
|---|---|---|---|---|
private_email | 1.000 | private_person | 0.909 | |
private_url | 1.000 | private_address | 0.922 | |
ip_address | 1.000 | account_number | 0.979 | |
private_date | 0.980 | personal_handle | 0.863 | |
private_phone | 0.993 |
pip install "transformers>=4.40" torch safetensors1import torch
2from transformers import AutoTokenizer, AutoModelForTokenClassification
3
4MODEL_ID = "FrameByFrame/korean-pii-e5-base"
5tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
6model = AutoModelForTokenClassification.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)
7model.eval()
8if torch.cuda.is_available():
9 model.cuda()민수씨 → 민수, 송파구에 → 송파구). The
benchmark numbers above include this normalization.1import re
2
3_TRAILING_JOSA = ["이에요","이라고","입니다","이야","이랑","한테","에게","으로","이가","이는",
4 "에서","이고","예요","씨","님","이","가","은","는","을","를","야","아","에","의","랑","께","고"]
5_DATE_END = re.compile(r".*(?:일|[0-9])", re.S)
6
7def _normalize(text, label, s, e):
8 while s < e and text[s] in " .,\t\n": s += 1
9 while e > s and text[e-1] in " .,\t\n": e -= 1
10 if label == "private_date":
11 m = _DATE_END.match(text[s:e])
12 if m and m.end() > 0: e = s + m.end()
13 elif label in ("private_person", "personal_handle", "private_address"):
14 for _ in range(2):
15 seg = text[s:e]
16 for j in _TRAILING_JOSA:
17 if seg.endswith(j) and (e - s) - len(j) >= 2:
18 e -= len(j); break
19 else:
20 break
21 return s, e
22
23def extract_pii(text: str, max_length: int = 256):
24 enc = tokenizer(text, truncation=True, max_length=max_length,
25 return_offsets_mapping=True, return_tensors="pt")
26 offsets = enc.pop("offset_mapping")[0].tolist()
27 with torch.no_grad():
28 logits = model(**{k: v.to(model.device) for k, v in enc.items()}).logits
29 pred = logits.argmax(-1)[0].tolist()
30 id2label = model.config.id2label
31
32 spans, active = [], None # active = [label, start, end]
33 for i, lid in enumerate(pred):
34 label = id2label[int(lid)]
35 cs, ce = offsets[i]
36 if cs == ce: # special token
37 if active: spans.append(active); active = None
38 continue
39 if label == "O":
40 if active: spans.append(active); active = None
41 continue
42 prefix, cat = label.split("-", 1)
43 if prefix in ("B", "S") or not active or active[0] != cat:
44 if active: spans.append(active)
45 active = [cat, cs, ce]
46 else:
47 active[2] = ce
48 if active: spans.append(active)
49
50 out = []
51 for cat, s, e in spans:
52 s, e = _normalize(text, cat, s, e)
53 if text[s:e].strip():
54 out.append({"label": cat, "start": s, "end": e, "text": text[s:e]})
55 return out1def redact(text: str) -> str:
2 spans = sorted(extract_pii(text), key=lambda s: s["start"], reverse=True)
3 for s in spans:
4 text = text[:s["start"]] + f"[{s['label'].upper()}]" + text[s["end"]:]
5 return text
6
7>>> redact("김민수님의 번호는 010-1234-5678입니다.")
8"[PRIVATE_PERSON]님의 번호는 [PRIVATE_PHONE]입니다."| field | description |
|---|---|
label | one of the 9 categories above |
start | character offset start (inclusive) |
end | character offset end (exclusive) |
text | the matched substring |
| Base model | intfloat/multilingual-e5-base (XLM-RoBERTa, ~278M) |
| Task | token classification, BIOES (9 PII classes → 37 labels) |
| Method | full fine-tune (token head randomly initialized; encoder fully trained) |
| Datasets | multi-domain Korean mix — KDPII (conversational, CC BY 4.0) + KLUE-NER person spans (news) + LLM-generated multi-domain documents (medical, legal, finance, e-commerce, HR, real-estate, social, gaming, IT, telecom, education, travel, delivery, email) with placeholder-filled PII + distribution-matched synthetic PII. All PII is synthetic/generated, never real. |
| Split | KDPII test held out (seed 42); 2 document domains (insurance, government) fully held out for unseen-domain eval; KLUE-val held out |
| Optimizer | AdamW, lr 3e-5, linear schedule, warmup 0.05 |
| Batch / seq | 32 per device, max_length 256 |
| Epochs | 3, best checkpoint by eval_span_f1 |
| Precision | bf16 |
| Hardware | 1× NVIDIA RTX A5000 |
personal_handle (~0.86 in-domain) is the weakest class — handles are open-vocabulary
(arbitrary usernames) and overlap with names; near its practical ceiling.extract_pii helper applies span normalization; if you decode logits yourself, apply
equivalent trimming to reproduce the reported numbers.intfloat/multilingual-e5-base (MIT). Training data includes KDPII (CC BY 4.0).1@misc{framebyframe-korean-pii-e5-base-2026,
2 title = {Korean PII (multilingual-e5-base): token classification for Korean PII},
3 author = {Mariappan, Vijayachandran},
4 year = {2026},
5 url = {https://huggingface.co/FrameByFrame/korean-pii-e5-base}
6}