m2v-e5-large-edu — static embeddings for educational content
A fast, torch-free static embedding model tuned for educational metadata. It is a
Model2Vec distillation of
intfloat/multilingual-e5-large
enriched with a domain vocabulary of curated education-domain keyword tags, so that
subject terms and multi-word phrases (e.g. "Quadratische Gleichung", "Erneuerbare
Energien") are represented as atomic units.
The base model is a multilingual XLM-RoBERTa-large, so the model still embeds many
languages; the added domain vocabulary is primarily German, so the strongest gains are
on German educational text. Inference is numpy-only — no PyTorch — and fast: in our CPU
benchmark it encoded ~7000 texts/s in a single process.
Distillation:model2vec.distill with
mean pooling (e5's native pooling — this choice has a large effect on quality),
pca_dims=512, float32, and SIF weighting (sif_coefficient=1e-4, Zipf-based frequency
down-weighting of common tokens).
Domain vocabulary: ~70,000 cleaned keyword tags (document frequency ≥ 3) extracted
from the keyword field of ~340k educational resources — mostly German subject terms —
added on top of the base subword vocabulary. Subword fallback is kept for out-of-vocabulary text.
Evaluation
Evaluated on a German educational subject-classification task and compared against other
embedding models.
Methodology (identical across models — only the embedding model differs): 25,068
educational items with 47 subject labels and a fixed train / validation / held-out test
split. Each embedding model is frozen; a small MLP classification head is trained on top
(Optuna hyperparameter search on validation), per-label decision thresholds are tuned on
validation, and F1 is reported on the held-out test split that is never used for training
or tuning (honest, non-optimistic evaluation).
What this shows: among static distillations of strong multilingual encoders, this model
lands in the same band (~0.56–0.60 macro) — the practical ceiling for frozen static
embeddings on this short, keyword-heavy metadata. A transformer (e5-small) stays slightly
ahead, as expected for static embeddings; the trade-off is that this model needs no
PyTorch and is far lighter and faster at inference. For this specific classification task
a plain TF-IDF + linear classifier scores higher still (macro ≈ 0.62), so the intended value
of this model is fast, torch-free semantic embeddings for the education domain, not being
the single best classifier for one dataset. These are domain-specific benchmark numbers,
not a general-purpose (e.g. MTEB) score.
Intended use
Fast embedding of (mostly German) educational metadata — titles, descriptions, keywords —
for classification, clustering, semantic search and retrieval, especially in CPU-only or
low-resource deployments where a transformer is too heavy.
Model2Vec passes a vocabulary through a
sentence-transformer, reduces dimensionality with PCA, and weights the token embeddings with
SIF weighting (frequent tokens down-weighted). At
inference it takes the (weighted) mean of the static token vectors of a text — no transformer
forward pass.
License
Inherits the base model's license (multilingual-e5-large: MIT).
Citation
@article{minishlab2024model2vec,
author = {Tulkens, Stephan and {van Dongen}, Thomas},
title = {Model2Vec: Fast State-of-the-Art Static Embeddings},
year = {2024},
url = {https://github.com/MinishLab/model2vec}
}
