You can deploy this model and start earning money today!

thealper2/
intfloat-multilingual-e5-base-tr-nli

Name: intfloat-multilingual-e5-base-tr-nli
Availability: InStock

Views

No views yet

e5-tr-nli — Turkish Sentence Embedding Model (Bi-Encoder)

A Turkish sentence-embedding (bi-encoder) model for retrieval and semantic search, fine-tuned from intfloat/multilingual-e5-base with contrastive learning (MultipleNegativesRankingLoss) on Turkish NLI triplets.

Embedding dimension: 768
Max sequence length: 256 tokens (trained); backbone supports 512
Pooling: mean pooling
Similarity function: cosine
Parameters: ~278M (XLM-RoBERTa-base backbone)
Prefix convention: e5-style query: / passage: (see Usage — required)

Intended Use
Usage
Training Data
Training Procedure
Evaluation
TR-MTEB Results
Limitations & Biases
Compute & Environmental Impact
Licensing
Citation
Reproduction

Intended Use

In scope

Turkish semantic search / passage retrieval (dense retrieval, RAG)
Semantic textual similarity, paraphrase & duplicate detection
Clustering and topic grouping of Turkish text
Feature extraction for downstream Turkish NLP classifiers

Out of scope

Re-ranking with query–document cross-attention (use a cross-encoder instead)
Long-document embedding beyond 256 tokens without chunking
High-stakes decisions (legal, medical, hiring) without human oversight
Non-Turkish text (backbone is multilingual, but this model is tuned for Turkish)

Usage

⚠️ e5 prefixes are mandatory

The model was trained with the e5 asymmetric prefixes and must be used with them, consistently at training / evaluation / inference:

Role	Prefix
Query / anchor / any symmetric-task sentence (STS, classification, clustering)	`query:`
Passage / document / positive & negative candidates	`passage:`

Omitting the prefixes degrades quality noticeably.

With `sentence-transformers`

python

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("thealper2/intfloat-multilingual-e5-base-tr-nli")

# Asymmetric retrieval: query vs. candidate passages
query = "query: Türkiye'nin başkenti neresidir?"
passages = [
    "passage: Ankara, Türkiye'nin başkentidir.",
    "passage: İstanbul Türkiye'nin en kalabalık şehridir.",
    "passage: Muz tropikal bir meyvedir.",
]
q = model.encode(query, normalize_embeddings=True)
p = model.encode(passages, normalize_embeddings=True)
print(cos_sim(q, p))   # highest score -> the Ankara passage

Semantic similarity (symmetric — use `query:` on both sides)

python

a = model.encode("query: Bugün hava çok güzel.", normalize_embeddings=True)
b = model.encode("query: Hava bugün oldukça güzel.", normalize_embeddings=True)
print(float(cos_sim(a, b)))   # ~0.9

With 🤗 Transformers (mean pooling)

python

import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("thealper2/intfloat-multilingual-e5-base-tr-nli")
mdl = AutoModel.from_pretrained("thealper2/intfloat-multilingual-e5-base-tr-nli")

def encode(texts):
    batch = tok(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
    with torch.no_grad():
        out = mdl(**batch)
    mask = batch["attention_mask"].unsqueeze(-1).float()
    emb = (out.last_hidden_state * mask).sum(1) / mask.sum(1)   # mean pooling
    return F.normalize(emb, p=2, dim=1)

emb = encode(["query: örnek cümle", "passage: örnek pasaj"])

Training Data

Source: mertcobanov/all-nli-triplets-turkish — a machine-translated Turkish version of the AllNLI (SNLI + MultiNLI) triplet set.
Format: (anchor, positive, negative) triplets, where the negative acts as a hard negative for the contrastive objective.
Column handling: Only the Turkish columns (anchor_translated, positive_translated, negative_translated) were used and renamed to anchor / positive / negative. All English columns were discarded.
Cleaning: rows with None / empty / whitespace-only fields were filtered out.
Resulting sizes (after filtering):

Split Triplets
train 277,167
dev 6,584
test 6,609

Training Procedure

Objective

Loss: CachedMultipleNegativesRankingLoss (MNRL with in-batch + hard negatives; the cached variant is used to simulate a large effective batch on limited VRAM).
Batch sampler: NO_DUPLICATES (required for MNRL to avoid trivial in-batch collisions).
In-batch negatives: every other positive/negative in the batch serves as a negative for a given anchor, so larger batches yield a stronger contrastive signal.

Hyperparameter search

An Optuna search (TPE sampler) was run before the final training:

Setting	Value
Trials	10
Subset	40,000 random training triplets
Steps / trial	600
Search space	`lr ∈ {1e-5, 2e-5, 3e-5}`, `batch ∈ {32, 64, 128}`, `warmup_ratio ∈ {0.0, 0.1}`
Selection metric	`stsb-tr` dev `spearman_cosine`
Best config	lr = 2e-5, batch = 32, warmup_ratio = 0.1 (dev spearman = 0.8285)

Final training

Setting	Value
Base model	`intfloat/multilingual-e5-base`
Epochs	1
Learning rate	2e-5
Batch size	32 (cached MNRL)
Warmup ratio	0.1
Max sequence length	256
Precision	bf16
Seed	42
Eval strategy	steps, monitoring `eval_stsb-tr-dev_spearman_cosine`
Checkpoint selection	`load_best_model_at_end=True` (best dev spearman)

Frameworks & hardware

sentence-transformers 5.2.2, transformers 5.0.0, torch 2.11.0+cu128, datasets 4.4.1, Python 3.12
1× NVIDIA GeForce RTX 5050 Laptop GPU (8 GB), CUDA 12.8

Evaluation

Baseline (base model) vs. Final (this model)

Measured on the dataset's own test triplets (TripletEvaluator, cosine accuracy) and on the external Turkish STS set emrecan/stsb-mt-turkish (EmbeddingSimilarityEvaluator, Spearman; scores normalised 0–5 → 0–1):

Metric	Baseline (e5-base)	Final	Δ
stsb-tr dev spearman_cosine	0.8092	0.8312	+0.0220
stsb-tr test spearman_cosine	0.7761	0.7814	+0.0053
nli-tr test cosine_accuracy	0.8968	0.9260	+0.0292

Methodology notes for TR-MTEB

Datasets pulled directly from HuggingFace (trmteb/*); metrics computed with a standalone script (no mteb package dependency), following standard MTEB protocols:
- STS → Spearman of cosine vs. gold; Retrieval → nDCG@10 / Recall@10 / MAP@10 via exact cosine search; Classification → logistic regression on frozen embeddings (accuracy + macro-F1); PairClassification → average precision of cosine; Clustering → V-measure (MiniBatchKMeans, k = #labels); BitextMining → top-1 nearest-neighbour accuracy.
e5 prefixes applied throughout (query: for queries/symmetric sides, passage: for corpus).
Retrieval search runs on-GPU in fp16.
75haber, thy_sa, irony-tr have no train split in the trmteb org, so a seeded 70/30 stratified self-split was used for those classification tasks (marked note in trmteb_results.json). All other classification sets use their official train/test.

Category summary (macro averages)

Task type	# datasets	Avg. main metric
STS (spearman_cosine)	1	0.791
BitextMining (accuracy)	1	0.975
Classification (accuracy)	6	0.817
PairClassification (AP)	3	0.573
Clustering (v-measure)	2	0.487
Retrieval (nDCG@10)	10	0.407
Overall (mean of category means)	23	0.675

TR-MTEB Results (full)

Evaluated on the TR-MTEB datasets (Baysan & Güngör, TR-MTEB, Findings of EMNLP 2025).

Task type	Dataset	Main metric	Value
STS	stsb-tr	spearman_cosine	0.7907
BitextMining	wmt16_en_tr	accuracy	0.9753
Classification	news-cat	accuracy	0.9600
Classification	75haber ᵃ	accuracy	0.8696
Classification	ts_timeline_news_category	accuracy	0.7250
Classification	thy_sa ᵃ	accuracy	0.8386
Classification	offenseval	accuracy	0.8427
Classification	irony-tr ᵃ	accuracy	0.6667
PairClassification	snli_tr	ap_cosine	0.5280
PairClassification	multinli_tr	ap_cosine	0.6324
PairClassification	xnli_tr	ap_cosine	0.5591
Clustering	ts_abstract_corpus	v_measure	0.5762
Clustering	630koseyazisi	v_measure	0.3978
Retrieval	tquad	ndcg@10	0.8222
Retrieval	quora-tr	ndcg@10	0.7323
Retrieval	squad-tr	ndcg@10	0.6584
Retrieval	scifact-tr	ndcg@10	0.5309
Retrieval	cqadupstack-gaming-tr	ndcg@10	0.3584
Retrieval	msmarco-tr	ndcg@10	0.2734
Retrieval	arguana-tr	ndcg@10	0.2583
Retrieval	nfcorpus-tr	ndcg@10	0.2272
Retrieval	fiqa-tr	ndcg@10	0.1237
Retrieval	scidocs-tr	ndcg@10	0.0887

Limitations & Biases

Translated training data. The NLI triplets are machine-translated; translation artefacts and noise are inherited, which can cap fine-grained semantic precision.
Short-text bias. Trained on NLI-style short sentences at 256 tokens; long-document retrieval requires chunking and may underperform (see the lower fiqa/scidocs scores).
Prefix sensitivity. Using the model without query:/passage: prefixes degrades results.
Domain gaps. Scientific/financial retrieval (scidocs, fiqa) is weak; the model is strongest on general-domain QA-style retrieval (tquad, squad-tr, quora-tr).
Societal bias. The backbone and NLI data may encode social biases; validate before deployment in sensitive settings.
Single epoch. Trained for 1 epoch; longer schedules or larger effective batches may improve retrieval further.

Compute & Environmental Impact

Single consumer laptop GPU (RTX 5050, 8 GB). Full pipeline (Optuna sweep + 1-epoch training on 277k triplets + evaluation) ran in a few GPU-hours. No multi-GPU/cluster training was used.

Licensing

Backbone intfloat/multilingual-e5-base: MIT.
Training data derives from AllNLI (SNLI is CC BY-SA 4.0; MultiNLI has mixed source licenses), translated to Turkish. Because the most restrictive component is CC BY-SA 4.0, this model card is released under CC BY-SA 4.0; verify that this suits your use case and comply with the ShareAlike terms.
Evaluation datasets belong to their respective authors (see the trmteb org and emrecan/stsb-mt-turkish).

Citation

This model:

bibtex

@misc{e5-tr-nli,
  title  = {e5-tr-nli: A Turkish Sentence Embedding Model},
  note   = {Fine-tuned from intfloat/multilingual-e5-base on Turkish NLI triplets},
  year   = {2026}
}

TR-MTEB benchmark:

bibtex

@inproceedings{baysan-gungor-2025-trmteb,
  title     = {{TR-MTEB}: A Comprehensive Benchmark and Embedding Model Suite for {T}urkish Sentence Representations},
  author    = {Baysan, Mehmet Selman and G{\"u}ng{\"o}r, Tunga},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year      = {2025}
}

Base model (E5):

bibtex

@article{wang2024multilingual,
  title   = {Multilingual E5 Text Embeddings: A Technical Report},
  author  = {Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal = {arXiv preprint arXiv:2402.05672},
  year    = {2024}
}

Reproduction

The full pipeline (data prep, Optuna sweep, training, and TR-MTEB evaluation) is scripted:

train_embedding_tr.py — --mode sweep | train | eval | mteb
eval_trmteb_hf.py — TR-MTEB evaluation straight from HuggingFace trmteb/* datasets
config.py, Makefile, requirements.txt

To publish (disabled by default — no secrets are used):

python

from sentence_transformers import SentenceTransformer
SentenceTransformer("models/e5-tr-nli-final").push_to_hub("thealper2/intfloat-multilingual-e5-base-tr-nli")

Split	Triplets
train	277,167
dev	6,584
test	6,609

thealper2/intfloat-multilingual-e5-base-tr-nli

e5-tr-nli — Turkish Sentence Embedding Model (Bi-Encoder)

Table of Contents

Intended Use

Usage

⚠️ e5 prefixes are mandatory

With sentence-transformers

Semantic similarity (symmetric — use query: on both sides)

With 🤗 Transformers (mean pooling)

Training Data

Training Procedure

Objective

Hyperparameter search

Final training

Frameworks & hardware

Evaluation

Baseline (base model) vs. Final (this model)

Methodology notes for TR-MTEB

Category summary (macro averages)

TR-MTEB Results (full)

Limitations & Biases

Compute & Environmental Impact

Licensing

Citation

Reproduction

thealper2/
intfloat-multilingual-e5-base-tr-nli

With `sentence-transformers`

Semantic similarity (symmetric — use `query:` on both sides)