You can deploy this model and start earning money today!

thivy/
norbert4-base-splade-finetuned-scand

Name: norbert4-base-splade-finetuned-scand
Availability: InStock

Views

No views yet

SPLADE NorBERT4-base — Fine-tuned on Scandinavian Multi-dataset

A SPLADE sparse retrieval model fine-tuned from ltg/norbert4-base (149M parameters, 51.2K vocabulary) on Norwegian, Danish, and Swedish datasets.

Model Description

Architecture: Regular SPLADE with MLMTransformer + SpladePooling (max pooling, ReLU activation)

Base Model: ltg/norbert4-base - A GPT-BERT hybrid trained on Scandinavian text

Training:

Duration: 1 epoch, 22,773 steps
Batch size: 8
Learning rate: 2e-5
Precision: BF16
Hardware: NVIDIA H100 NVL

Loss Function: SpladeLoss (SparseMultipleNegativesRankingLoss + FlopsLoss)

Query regularizer weight: 1e-4
Document regularizer weight: 3e-3

Datasets (round-robin sampling):

Fremtind/all-nli-norwegian - Norwegian NLI
DDSC/nordic-embedding-training-data - Multi-language tasks
ScandiQA, NorQuAD, OpenBookQA - Question answering
PAWS-X Norwegian - Paraphrase detection

Known Issue: 0% Metric Sparsity

⚠️ The sparsity metric reports 0% despite the model being functionally sparse.

Why this happens:

NorBERT4's MLM head applies: 30 * sigmoid(x/7.5), forcing all logits to (0, 30) range
SPLADE's ReLU activation: ReLU(log(1+exp(x))) - cannot produce zeros from strictly positive values
Result: The metric shows all 51,200 dimensions active, but many have very small weights

This is not a bug. The model works correctly. It just needs a threshold at inference time.

Verification Tests

The model was tested with Norwegian queries to verify it works correctly:

Test 1: Retrieval Accuracy

Query: "Hva er hovedstaden i Norge?" (What is the capital of Norway?)

✓ Correct doc (Oslo):    20.78
✗ Wrong docs:            10.36 (Bergen), 1.85 (other)
Margin: +10.42

Query: "Hvem vant fotball-VM i 2022?" (Who won the World Cup in 2022?)

✓ Correct doc (Argentina): 18.33
✗ Wrong docs:              2.48 (other), 2.47 (other)
Margin: +15.85

Query: "Hva er symptomene på influensa?" (What are symptoms of the flu?)

✓ Correct doc (Influenza): 20.66
✗ Wrong docs:              1.70 (other), 1.64 (other)
Margin: +18.96

Result: Perfect retrieval with all correct documents ranked first

Test 2: Semantic Token Expansion

The model learns to expand queries to semantically related terms:

Query: "hovedstaden i Norge"

Top tokens: hovedstaden(3.21), Norge(2.92), Oslo(2.39), Noreg(2.30), 
            capital(1.08), hovedstad(2.66), Bergen(1.05)

Query: "fotball-VM"

Top tokens: -VM(2.51), fotball(2.21), VM(2.02), FIFA(1.42), 
            Championships(1.40), UEFA(1.34), mesterskap(0.84)

Query: "influensa symptomene"

Top tokens: influensa(3.06), symptomene(2.44), symptomer(2.12), 
            feber(0.74), forkjølelse(1.00)

Test 3: Sparsity with Threshold

Applying threshold=0.05 to zero out small values:

Threshold	Query Active	Query Sparsity	Doc Active	Doc Sparsity
None	51,200	0%	51,200	0%
0.05	469	99.1%	166	99.7%
0.1	181	99.6%	75	99.9%

Result: Can achieve >99% sparsity while maintaining perfect retrieval

How to Use

Basic Usage

python

from sentence_transformers import SparseEncoder

model = SparseEncoder("thivy/norbert4-base-splade-finetuned-scand", trust_remote_code=True)

queries = ["Hva er hovedstaden i Norge?"]
documents = [
    "Oslo er hovedstaden og den størst byen i Norge.",
    "Bergen er en vakker by på vestlandet.",
]

query_embeddings = model.encode(queries)
doc_embeddings = model.encode(documents)

# Compute similarity
from sentence_transformers.util import cos_sim
scores = cos_sim(query_embeddings, doc_embeddings)
print(scores)

With Threshold (Recommended)

To achieve high sparsity, apply threshold at inference:

python

from sentence_transformers import SparseEncoder

model = SparseEncoder("thivy/norbert4-base-splade-finetuned-scand", trust_remote_code=True)

texts = ["Hva er hovedstaden i Norge?"]
embeddings = model.encode(texts, convert_to_sparse_tensor=False)

# Apply threshold to get ~99% sparse embeddings
threshold = 0.05
embeddings[embeddings < threshold] = 0

print(f"Active dimensions: {(embeddings > 0).sum().item()}/51200")

Verification Script

Run this to verify the model works on your setup:

python

from sentence_transformers import SparseEncoder
import torch
import numpy as np

model = SparseEncoder('thivy/norbert4-base-splade-finetuned-scand', trust_remote_code=True)

queries = [
    'Hva er hovedstaden i Norge?',
    'Hvem vant fotball-VM i 2022?',
    'Hva er symptomene på influensa?',
]

documents = [
    'Oslo er hovedstaden og den mest folkerike byen i Norge.',
    'Argentina vant FIFA verdensmesterskapet i fotball i 2022.',
    'Influensa er en virussykdom som gir symptomer som feber, hoste.',
    'Bergen er en vakker by på vestlandet.',
    'Norsk bokmål og nynorsk er de to offisielle skriftspråkene i Norge.',
]

print('=== RAW EMBEDDINGS (no threshold) ===')
q_emb = model.encode_query(queries, convert_to_sparse_tensor=False).cpu().numpy()
d_emb = model.encode_document(documents, convert_to_sparse_tensor=False).cpu().numpy()

sims = q_emb @ d_emb.T
print('Query-Document Similarity (should have high diagonal):')
for i, q in enumerate(queries):
    best = np.argmax(sims[i])
    print(f'Q{i+1} best match: D{best+1} (score: {sims[i][best]:.2f})')

print('\n=== WITH THRESHOLD = 0.05 ===')
q_sparse = q_emb.copy()
d_sparse = d_emb.copy()
q_sparse[q_sparse < 0.05] = 0
d_sparse[d_sparse < 0.05] = 0

q_active = np.mean([np.count_nonzero(q_sparse[i]) for i in range(len(queries))])
d_active = np.mean([np.count_nonzero(d_sparse[i]) for i in range(len(documents))])

print(f'Query active dims: {q_active:.0f} / 51200 ({100*q_active/51200:.1f}%)')
print(f'Doc active dims: {d_active:.0f} / 51200 ({100*d_active/51200:.1f}%)')

sims_sparse = q_sparse @ d_sparse.T
print('Similarity with threshold (rankings should be same):')
for i, q in enumerate(queries):
    best = np.argmax(sims_sparse[i])
    print(f'Q{i+1} best match: D{best+1} (score: {sims_sparse[i][best]:.2f})')

Token Expansion Script

See which tokens get high weights in the embeddings:

python

from sentence_transformers import SparseEncoder

model = SparseEncoder('thivy/norbert4-base-splade-finetuned-scand', trust_remote_code=True)

queries = [
    'Hva er hovedstaden i Norge?',
    'Hvem vant fotball-VM i 2022?',
]

embeddings = model.encode(queries)
decoded = model.decode(embeddings, top_k=15)

for d, q in zip(decoded, queries):
    print(f'Query: {q}')
    tokens = ', '.join([f'{tok}({score:.2f})' for tok, score in d])
    print(f'Top tokens: {tokens}\n')

Comparison with Official SPLADE

This model uses the same SPLADE approach as the official distilbert example from MS MARCO, but:

Base model: NorBERT4 (Norwegian) vs DistilBERT (English)
Vocabulary: 51,200 vs 30,522
Datasets: Scandinavian multi-lingual vs English MS MARCO
Sparsity issue: NorBERT4's sigmoid head vs DistilBERT's direct logits

For future training without the sparsity metric issue, use thivy/norbert4-base-splade which has the sigmoid activation removed.

Technical Details

Sparse Tensor Output: The model outputs sparse tensors. Convert to dense for similarity computation:

embeddings = model.encode(texts, convert_to_sparse_tensor=False) # Get dense tensors

Memory Usage:

Model size: ~300MB
Inference VRAM: ~2GB for typical batch sizes
Embedding sparsity saves memory vs dense embeddings

Supported Languages:

Norwegian (primary)
Danish (in training data)
Swedish (in training data)

Original Base Model

Based on ltg/norbert4-base by the Language Technology Group at University of Oslo.

See the original model card for architecture details and training information.

Citation

If you use this model, please cite the relevant papers:

bibtex

@article{formal2021splade,
  title={SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking},
  author={Formal, Thibault and Lassance, Carlos and Randeaux, Benjamin and Piwowarski, Benjamin},
  journal={arXiv preprint arXiv:2107.05720},
  year={2021}
}

@inproceedings{papadopoulou2023norbert,
  title={NorBERT and NorT5—Norwegian BERT and T5 Models},
  author={Papadopoulou, Aikaterini and Recla, Álvaro Arroyo and others},
  booktitle={NLP4NLP Workshop @ ACL 2023},
  year={2023}
}

License

MIT