This model is designed to redact and classify Personally Identifiable Information (PII) from multilingual text. It has been fine-tuned on the open-pii-masking-500k-ai4privacy dataset and supports multiple languages, including French (fr), English (en), German (de), Telugu (te), Hindi (hi), Italian (it), Spanish (es), and Dutch (nl).

Evaluation Metrics

The table below summarizes the detailed evaluation results per PII label. Metrics are presented as percentages rounded to two decimal places. For the "O" (Non-PII) label, precision, recall, and F1 score are not applicable (n/a) due to the absence of true positives.

Label	TP	FP	FN	Accuracy	Precision	Recall	F1 Score
O (Non-PII)	0	734	0	98.97%	n/a	n/a	n/a
GIVENNAME	6623	661	352	86.73%	90.93%	94.95%	92.90%
SURNAME	2786	877	162	72.84%	76.06%	94.50%	84.28%
CITY	1763	216	225	79.99%	89.09%	88.68%	88.88%
DATE	2195	1	3	99.82%	99.95%	99.86%	99.91%
AGE	176	7	2	95.14%	96.17%	98.88%	97.51%
EMAIL	2981	0	0	100.0%	100.0%	100.0%	100.0%
CREDITCARDNUMBER	601	57	35	86.72%	91.34%	94.50%	92.89%
SEX	103	45	1	69.13%	69.59%	99.04%	81.75%
SOCIALNUM	364	134	20	70.27%	73.09%	94.79%	82.54%
TIME	1631	1	3	99.76%	99.94%	99.82%	99.88%
TELEPHONENUM	3537	10	9	99.47%	99.72%	99.75%	99.73%
IDCARDNUM	1540	314	148	76.92%	83.06%	91.23%	86.96%
ZIPCODE	311	39	16	84.97%	88.86%	95.11%	91.87%
DRIVERLICENSENUM	296	143	26	63.66%	67.43%	91.93%	77.79%
PASSPORTNUM	482	285	25	60.86%	62.84%	95.07%	75.67%
TITLE	224	68	78	60.54%	76.71%	74.17%	75.42%
BUILDINGNUM	292	45	14	83.19%	86.65%	95.42%	90.85%
STREET	1272	155	67	85.14%	89.14%	94.99%	91.97%
TAXNUM	471	101	34	77.72%	82.34%	93.27%	87.47%
GENDER	123	35	9	73.65%	77.85%	93.18%	84.83%

Overall Evaluation

Accuracy: 95.03%
Precision: 87.61%
Recall: 95.76%
F1 Score: 91.50%
Total True Positives (TP): 27,771
Total False Positives (FP): 3,928
Total False Negatives (FN): 1,229

Macro-Averaged Metrics

Accuracy: 82.17%
Precision: 80.99%
Recall: 89.96%
F1 Score: 84.91%

Model Behavior & Limitations

Evaluation Focus:
The metrics above reflect performance on the test split of the open-pii-masking-500k-ai4privacy dataset. This model both redacts and classifies PII into specific categories (e.g., GIVENNAME, EMAIL). Real-world performance may vary depending on text domain and language, so additional validation is recommended. For support, contact support@ai4privacy.com.
Strengths:
- High recall (95.76%) ensures most PII is detected.
- Exceptional performance on labels like "EMAIL" (100% F1), "DATE" (99.91% F1), and "TIME" (99.88% F1).
Limitations:
- Lower precision for labels such as "PASSPORTNUM" (62.84%) and "DRIVERLICENSENUM" (67.43%), indicating a higher rate of false positives.
- The "O" (Non-PII) label has no true positives, making precision, recall, and F1 score not applicable (n/a).

Disclaimer

This model card details the evaluation metrics and fine-tuning parameters for the multilingual anonymiser with PII classification capabilities. Please note:

The model is provided as-is under the MIT License.
It is intended for both redaction and PII classification purposes.
Users should thoroughly test and evaluate its performance on their specific datasets before deploying in production environments.

Ai4Privacy – Committed to protecting personal data in the age of AI.

ai4privacy/llama-ai4privacy-multilingual-categorical-anonymiser-openpii

Multilingual Anonymiser OpenPII (Ai4Privacy)

Evaluation Metrics

Overall Evaluation

Macro-Averaged Metrics

Model Behavior & Limitations

Disclaimer

ai4privacy/
llama-ai4privacy-multilingual-categorical-anonymiser-openpii