Strategy: ROUND_ROBIN multi-dataset sampling (prevents catastrophic forgetting)
Batch Size: 16 (effective batch size: 32 with gradient accumulation)
Learning Rate: 5e-6 (low LR to prevent overfitting)
Epochs: 1 epoch through all datasets
Loss: MultipleNegativesRankingLoss
Early Stopping: Tracks average loss across all three datasets
Regularization: weight_decay=0.015, no warmup

Performance

MTEB Retrieval Benchmarks (vs previous models):

Task	Multi-Dataset	QA-only	NLI-only	Improvement
NorQuadRetrieval (ndcg@10)	0.232	0.209	0.163	+11.0%
SNLRetrieval (ndcg@10)	0.818	0.765	0.519	+6.9%

Usage

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("thivy/norbert4-base-scandinavian-embedding")

# Encode sentences
sentences = [
    "Dette er en norsk setning",
    "Detta är en svensk mening", 
    "Dette er en dansk sætning"
]

embeddings = model.encode(sentences)

# Compute similarity
from sentence_transformers.util import cos_sim
similarity = cos_sim(embeddings[0], embeddings[1])

Intended Use

Semantic search across Scandinavian languages
Document retrieval and ranking
Question-answering systems
Cross-lingual similarity (NO/DA/SV)
Text clustering and classification

Limitations

Max sequence length: 256 tokens (longer texts are truncated)
Optimized for Scandinavian languages (Norwegian, Danish, Swedish)
Best performance on retrieval tasks (not instruction-following)

Citation

If you use this model, please cite:

bibtex

@misc{norbert4-scandi-embedding,
  title={NorBERT4 Scandinavian Embedding Model},
  author={Thivyesh Ahilathasan},
  year={2025},
  url={https://huggingface.co/thivy/norbert4-base-scandinavian-embedding}
}

Related Models

Base: ltg/norbert4-base
Large: thivy/norbert4-large-scandinavian-embedding (coming soon)