pii-redactor-tokenizer-v1
Byte-level BPE tokenizer for the PII_MODEL browser-local redaction model.
Configuration
- tokenizer type: byte-level BPE
- vocab size: 64000
- special tokens: 21
- target corpus characters: 5000000000
- accepted corpus characters: 4376978638
- accepted documents: 5474812
This tokenizer is intended for span-level PII detection and placeholder-aware denoising.
