Swedish contrastive fine-tune of BAAI/bge-m3. Beats KBLab/sentence-bert-swedish-cased on both SuperLim-2 tasks that card published: SweParaphrase (STS) and SweFAQ (in-category retrieval).

Result: vs vanilla BGE-M3, this checkpoint improves SuperLim SweParaphrase Spearman by +2.3% (0.8165 → 0.8354) and SweFAQ in-category accuracy by +7.6% (0.6055 → 0.6514). Clean Swedish retrieval (SwednRetrieval nDCG@10, no train overlap) is +6.5% (0.7335 → 0.7809). Nordic MTEB mean is only +1.5%; Swedish classification dropped vs the base (SweRec −0.047, SwedishSentiment −0.028). Not a “multilingual hold.”

Related: oxfrug/qwen3-embedding-0.6b-swedish-superlim — Qwen-family 0.6B on the same protocol.

Model details

	Base	This
Base model	`BAAI/bge-m3`	same architecture, Swedish mix
Relation	—	fine-tune
Params	568M	568M
Embedding dim	1024 (Matryoshka)	768 recommended (truncate + re-L2)
Library	sentence-transformers	sentence-transformers
License	MIT	MIT

The 768-d cut keeps the STS win and lifts SweFAQ to the Qwen3-Embedding-8B zero-shot number.

Quickstart

pip install -U sentence-transformers

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("oxfrug/bge-m3-swedish-superlim")
emb = model.encode(["Ett plan lyfter.", "Ett flygplan lyfter."], normalize_embeddings=True)
print(float(emb[0] @ emb[1]))

Retrieval at the recommended 768-d cut (query prefix + Matryoshka truncate):

python

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("oxfrug/bge-m3-swedish-superlim")
QUERY = "Represent this sentence for searching relevant passages: "

def embed(texts, query=False, dim=768):
    texts = [QUERY + t if query else t for t in texts]
    vec = model.encode(texts, normalize_embeddings=True)
    if dim and vec.shape[1] > dim:
        vec = vec[:, :dim]
        vec /= np.clip(np.linalg.norm(vec, axis=1, keepdims=True), 1e-12, None)
    return vec

q = embed(["När får jag föräldrapenning?"], query=True)
d = embed(["Du kan ansöka på Mina sidor."], query=False)
print(float(q[0] @ d[0]))

STS / clustering: no prefix. Retrieval: prefix the query only.

Benchmarks

All “this” numbers below are @768 (first 768 dims, re-L2). Base is vanilla BAAI/bge-m3 on the same protocol. Δ is this − base.

SuperLim-2 (held-out test)

Spearman of cosine vs human 0–5 on SweParaphrase (n=1378). SweFAQ is in-category accuracy (n=109). Same frozen sbx/superlim-2 splits as the KBLab re-run.

Model	Params	SweParaphrase ρ	Δ vs BGE	SweFAQ	Δ vs BGE
KBLab v2.0 (card)	110M	0.8213	—	0.5871	—
KBLab v2.0 (our re-run)	110M	0.8207	—	0.5780	—
BGE-M3 zero-shot (base)	568M	0.8165	—	0.6055	—
this @768 (recommended)	568M	0.8354	+0.0189	0.6514	+0.0459
this @1024	568M	0.8353	+0.0188	0.6422	+0.0367

SweParaphrase test was never in train. SweFAQ used the official train split (plus Swedish healthcare Q–A). Do not call SweFAQ zero-shot. SweFAQ n=109 is +8 correct vs the KBLab re-run; treat it as a small test.

Swedish tasks — MTEB (Scandinavian, v1)

mteb 2.19.3, max_seq_length=512 (KBLab 256). Official SweFAQ score is nDCG@10, not SuperLim in-category accuracy.

Task	BGE-M3 (base)	this @768	Δ	Note
DalajClassification	0.4979	0.4993	+0.0014	no train overlap
SweRecClassification	0.8121	0.7647	−0.0474	no train overlap; regressed
SwedishSentimentClassification	0.9572	0.9293	−0.0279	no train overlap; regressed
SweFaqRetrieval nDCG@10	0.7170	0.8221	+0.1051	† supervised (SweFAQ-train)
SwednRetrieval	0.7335	0.7809	+0.0474	no train overlap; clean retrieval
SwednClusteringP2P	0.3654	0.3554	−0.0100	no train overlap
SwednClusteringS2S	0.1483	0.1991	+0.0508	no train overlap
MassiveIntent (sv)	0.6915	0.6824	−0.0091	no train overlap
MassiveScenario (sv)	0.7494	0.7553	+0.0059	no train overlap
SV mean (9)	0.6303	0.6432	+0.0129

† Supervised for this model. SwednRetrieval is the number to quote for Swedish retrieval generalization.

KBLab on the same nine tasks is in the dump; SV mean 0.6132. This mix beats KBLab on the Swedish mean, but that is not the before/after vs BGE.

Nordic category means (27/28 tasks)

DKHateClassification gated, skipped.

	BGE-M3 (base)	this @768	Δ
Classification (12)	0.6115	0.5960	−0.0155
Retrieval (7)	0.6424	0.6915	+0.0491
Clustering (6)	0.3953	0.4101	+0.0148
Bitext (2)	0.6924	0.6876	−0.0048
Overall (27)	0.5774	0.5862	+0.0088

The mix is a retrieval specialist. Classification and a bit of bitext were spent to buy SweFAQ / Swedn. Not a general “best Nordic embedder.”

Raw dumps: superlim/test.json, mteb/MTEB_Scandinavian_v1/. These files document the run; they do not put the model on the official MTEB leaderboard.

Why it moved

Training is MNRL on high-similarity Swedish pairs (SweParaphrase ≥ 4) plus FAQ / healthcare query–passage pairs with the BGE retrieval prefix. That is the same geometry SuperLim SweParaphrase and SweFAQ / SwednRetrieval reward, so those go up. Label-driven classification (SweRec, SwedishSentiment) was not in the loss; the contrastive mix pulled those embeddings around and they dropped vs the already-strong multilingual BGE baseline. SwednClusteringS2S rose; P2P did not — short-text clustering is not the same as FAQ retrieval.

How it was made

Dataset composition and licensing

Weights are MIT (same as BGE-M3). Training text is not MIT. SuperLim pieces are CC BY 4.0 (Språkbanken). Healthcare Q–A is an extra in-house mix; source file / license was not checked into this repo — treat as unknown, do not redistribute that split from here.

Source	Split used	Role	Approx. size	License
SuperLim-2 SweParaphrase	train only, pairs with score ≥ 4	STS positives for MNRL	subset of ~8.6k pair set	CC BY 4.0
SuperLim-2 SweNLI	train, entailment only	extra positives	40k (as trained)	CC BY 4.0
SuperLim-2 SweFAQ	train Q–A, BGE query prefix	retrieval	small (full SweFAQ ~1k rows)	CC BY 4.0
Swedish healthcare Q–A	train Q–A, BGE query prefix	retrieval	not published here	unknown

SuperLim dev/test strings were blocked. SweFAQ test is therefore not zero-shot. Swedn / SweRec / Dalaj / SwedishSentiment were not in train.

Hub dataset: sbx/superlim-2 (sweparaphrase, swenli, swefaq).

Training configuration (what is known)


Start	vanilla `BAAI/bge-m3`
Loss	MultipleNegativesRankingLoss
Epochs	1
Precision	bf16, full fine-tune (not LoRA)
Query prefix (retrieval pairs only)	`Represent this sentence for searching relevant passages:`
Ship dim	768 (Matryoshka truncate + re-L2)
Dev/test leak	SuperLim dev/test strings blocked

Not in this repo (original llm-lab training tree is not published here): learning rate, batch size, warmup, max sequence length during train, seed, exact pair counts after filtering, wall-clock, GPU SKU. Do not invent those. 4-bit NF4 after the fact gives the STS win back (0.820); ship bf16.

Reproducing evaluation

MTEB Scandinavian as on this card:

bash

pip install "mteb==2.19.3" sentence-transformers
python - <<'PY'
import mteb
from sentence_transformers import SentenceTransformer

class BgeSv768(SentenceTransformer):
    """First-768 Matryoshka cut + re-L2, BGE query prefix on retrieval."""
    pass

# Card numbers used max_seq_length=512, encode batch_size=32,
# normalize_embeddings=True, skipped DKHateClassification.
# Raw payloads: mteb/MTEB_Scandinavian_v1/ on this repo.
print("Use the JSON dump to match the card; a fresh mteb run needs the same wrapper.")
PY

The JSON under mteb/MTEB_Scandinavian_v1/ is the actual mteb.evaluate output (mteb 2.19.3, torch 2.13, sentence-transformers 5.7, transformers 5.14.1). SuperLim protocol: Spearman of cosine vs 0–5 on SweParaphrase-test; SweFAQ in-category accuracy on 109 items, query prefix on, dim 768. Dump: superlim/test.json.

A bit-identical training script is not in this repository.

Intended use

Swedish STS, clustering, and retrieval. Recommended embedding size is 768. Prefer this over vanilla BGE-M3 when SweFAQ / Swedn-style retrieval matters. Prefer vanilla BGE-M3 when SweRec / SwedishSentiment-style classification is the job.

Limits

568M / ~1.1 GB bf16. Slower to encode than 110M KBLab.
SweFAQ test is 109 items. @768 is +8 correct vs the KBLab re-run — noisy.
SuperLim claim is SweParaphrase + SweFAQ only.
Known failure vs base BGE-M3: SweRecClassification (−0.047), SwedishSentimentClassification (−0.028), Nordic classification mean (−0.016), SwednClusteringP2P (−0.010), MassiveIntent sv (−0.009).
SweFAQ nDCG@10 on MTEB is supervised. Quote SwednRetrieval for generalization.
4-bit NF4 gives the STS win back (0.820). Ship bf16.
Healthcare Q–A license unknown; SuperLim train licenses are CC BY 4.0 — the weights stay MIT, the data does not.

License

MIT, same as BGE-M3. Fine-tune by oxfrug. SuperLim training data is CC BY 4.0 (Språkbanken); cite SuperLim if you redistribute derivatives of that split.

Citation

bibtex

@misc{bge-m3,
  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng},
  year={2024},
  eprint={2402.03216},
  archivePrefix={arXiv}
}

@inproceedings{berdicevskis-etal-2023-superlim,
  title = {Superlim: A {S}wedish Language Understanding Evaluation Benchmark},
  author = {Berdicevskis, Aleksandrs and Bouma, Gerlof and Kurtz, Robin and Morger, Felix and {\"O}hman, Joey and Adesam, Yvonne and Borin, Lars and Dann{\'e}lls, Dana and Forsberg, Markus and Isbister, Tim and Lindahl, Anna and Malmsten, Martin and Rekathati, Faton and Sahlgren, Magnus and Volodina, Elena and B{\"o}rjeson, Love and Hengchen, Simon and Tahmasebi, Nina},
  booktitle = {Proceedings of EMNLP 2023},
  year = {2023}
}