e5-sk-large is a Slovak text embedding model (365M parameters, 1024-dimensional embeddings) built by applying vocabulary trimming and fine-tuning to
multilingual-e5-large. It achieves competitive performance with proprietary embedding APIs on
SkMTEB — the first comprehensive Slovak text embedding benchmark — while being 35% smaller than the original model and fully locally deployable.
For a smaller, faster variant, see
e5-sk-small (45M parameters).
Then you can load this model and run inference.
1from sentence_transformers import SentenceTransformer
2
3model = SentenceTransformer("slovak-nlp/e5-sk-large")
4
5# Retrieval
6query_embedding = model.encode("query: Čo je hlavné mesto Slovenska?")
7passage_embedding = model.encode("passage: Bratislava je hlavné a najväčšie mesto Slovenska.")
8similarity = model.similarity(query_embedding, passage_embedding)
9print(similarity) # tensor([[0.9269]])
10
11# Batch encoding
12sentences = [
13 "query: Aké je počasie v Bratislave?",
14 "passage: V Bratislave je dnes slnečno a teplo.",
15 "passage: Bratislava leží na brehu Dunaja.",
16]
17embeddings = model.encode(sentences)
18print(embeddings.shape) # (3, 1024)
1import torch
2import torch.nn.functional as F
3from transformers import AutoTokenizer, AutoModel
4
5def average_pool(last_hidden_states, attention_mask):
6 last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
7 return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
8
9tokenizer = AutoTokenizer.from_pretrained("slovak-nlp/e5-sk-large")
10model = AutoModel.from_pretrained("slovak-nlp/e5-sk-large")
11
12texts = [
13 "query: Čo je hlavné mesto Slovenska?",
14 "passage: Bratislava je hlavné a najväčšie mesto Slovenska.",
15]
16
17batch_dict = tokenizer(texts, max_length=512, padding=True, truncation=True, return_tensors="pt")
18with torch.no_grad():
19 outputs = model(**batch_dict)
20
21embeddings = average_pool(outputs.last_hidden_state, batch_dict["attention_mask"])
22embeddings = F.normalize(embeddings, p=2, dim=1)
23print((embeddings[0] @ embeddings[1]).item())
Step 1 — Vocabulary Trimming. Before fine-tuning,
Vocabulary Trimming (Ushio et al., 2023) was applied to
multilingual-e5-large to remove tokens irrelevant to Slovak.
Token frequencies were computed on
FineWeb2-Slovak, a quality-filtered Slovak web corpus, and the top 60K tokens (out of 250K) were retained. This reduced the model from 560M to
365M parameters (35% reduction) without meaningful performance loss.
Step 2 — Fine-tuning. The trimmed model was fine-tuned on curated Slovak datasets from the
skLEP benchmark:
Evaluated on
SkMTEB — 31 datasets across 7 task types. Scores are percentages (higher is better).
1@inproceedings{suppa2025skmteb,
2 title = {{SkMTEB}: {Slovak} Massive Text Embedding Benchmark and Model Adaptation},
3 author = {{\v{S}}uppa, Marek and Ridzik, Andrej and Hl{\'a}dek, Daniel and
4 Kna{\v{z}}ekov{\'a}, Nat{\'a}lia and Ondrejov{\'a}, Vikt{\'o}ria},
5 year = {2025},
6 eprint = {2606.13647},
7 archivePrefix = {arXiv},
8 url = {https://arxiv.org/abs/2606.13647}
9}
10
11@inproceedings{reimers-2019-sentence-bert,
12 title = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
13 author = {Reimers, Nils and Gurevych, Iryna},
14 booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
15 year = {2019},
16 publisher = {Association for Computational Linguistics},
17 url = {https://arxiv.org/abs/1908.10084}
18}
