Views
No views yet
1from sentence_transformers import SentenceTransformer
2import numpy as np
3
4model = SentenceTransformer("sentence-transformers/LaBSE")
5a = "Алғашқы мәтін"
6b = "Парафраз мәтіні"
7emb_a = model.encode(a, normalize_embeddings=True)
8emb_b = model.encode(b, normalize_embeddings=True)
9sim = np.dot(emb_a, emb_b)
10is_duplicate = sim >= 0.7
11print(f"Similarity: {sim:.3f}, Duplicate: {is_duplicate}")
12
13
14
15## Training & Evaluation
16
17Dataset: Arailym-tleubayeva/KazakhTextDuplicates
18Metrics: Accuracy, Precision, Recall, F1, ROC‑AUC
19Validation split: Stratified 90/10
20Threshold tuning: Grid search on validation for best F1.
21Metric Value
22Accuracy 0.94
23Precision 1.00
24Recall 0.94
25F1‑score 0.97
26
27
28@misc{tleubayeva2025kazakhtextduplicates,
29 author = {Тлеубаева, А.О.},
30 title = {KazakhTextDuplicates: Near‑duplicate Detection Model},
31 year = {2025},
32 howpublished = {\url{https://huggingface.co/Arailym-tleubayeva/KazakhTextDuplicates-Detector}}
33}