Views
No views yet
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 via layer pruning + vocabulary pruning.| Property | Value |
|---|---|
| Teacher | sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 |
| Architecture | MiniLM-L12 (pruned) |
| Hidden dim | 384 |
| Layers | 6 / 12 |
| Layer indices | [0, 2, 4, 7, 9, 11] |
| Strategy | 6 layers, evenly spaced (general-purpose) |
| Parameters | 106,825,344 |
| Model size (FP32) | 98.1MB |
| Distilled | No |
==============================================================
TEACHER: MiniLM-L12 → STUDENT: 6L / 38,775 vocab
==============================================================
TEACHER STUDENT
─────────────────────────── ───────────────────────────
┌─────────────────────────┐ ┌─────────────────────────┐
│ Input Tokens │ │ Input Tokens │
└────────────┬────────────┘ └────────────┬────────────┘
│ │
┌────────────┴────────────┐ ┌────────────┴────────────┐
│ Embeddings │ │ Embeddings (pruned) │
│ vocab: 250,002 │ │ vocab: 38,775 │
│ dim: 384 │ │ dim: 384 │
└────────────┬────────────┘ └────────────┬────────────┘
│ │
┌─────────────────────────┐ ┌─────────────────────────┐
│ Layer 0 │ ──► │ Layer 0 ← L0 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 1 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 2 │ ──► │ Layer 1 ← L2 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 3 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 4 │ ──► │ Layer 2 ← L4 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 5 │ ╳ │ │
├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┤ │ │
│ Layer 6 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 7 │ ──► │ Layer 3 ← L7 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 8 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 9 │ ──► │ Layer 4 ← L9 │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 10 │ ╳ │ │
├─────────────────────────┤ ├─────────────────────────┤
│ Layer 11 │ ──► │ Layer 5 ← L11 │
└────────────┬────────────┘ └────────────┬────────────┘
│ │
┌────────────┴────────────┐ ┌────────────┴────────────┐
│ Mean Pooling │ │ Mean Pooling │
│ → 384d embedding │ │ → 384d embedding │
└─────────────────────────┘ └─────────────────────────┘
Size: 448.0MB (FP32) → 98.1MB (FP32)
Params: 117,451,392 → 25,714,176
Reduction: 78.1%
==============================================================1from sentence_transformers import SentenceTransformer
2
3model = SentenceTransformer("L6_uniform", trust_remote_code=True)
4
5sentences = [
6 "Hello, how are you?",
7 "안녕하세요",
8 "Bonjour, comment allez-vous?",
9]
10
11embeddings = model.encode(sentences)
12print(embeddings.shape) # (3, 384)| Task Group | Average |
|---|---|
| Classification | 58.9% |
| Clustering | 34.38% |
| STS | 64.74% |
| Task | Average | Details |
|---|---|---|
| AmazonCounterfactualClassification | 68.27% | de: 69.95%, en: 69.4%, en-ext: 68.78% |
| Banking77Classification | 73.53% | default: 73.53% |
| ImdbClassification | 60.64% | default: 60.64% |
| MTOPDomainClassification | 75.11% | en: 84.23%, es: 78.09%, th: 76.25% |
| MassiveIntentClassification | 37.62% | en: 62.82%, zh-CN: 60.31%, ja: 58.99% |
| MassiveScenarioClassification | 41.45% | en: 68.66%, zh-CN: 67.6%, ja: 63.51% |
| ToxicConversationsClassification | 61.36% | default: 61.36% |
| TweetSentimentExtractionClassification | 53.21% | default: 53.21% |
| Task | Average | Details |
|---|---|---|
| ArXivHierarchicalClusteringP2P | 50.12% | default: 50.12% |
| ArXivHierarchicalClusteringS2S | 46.66% | default: 46.66% |
| BiorxivClusteringP2P.v2 | 25.42% | default: 25.42% |
| MedrxivClusteringP2P.v2 | 28.32% | default: 28.32% |
| MedrxivClusteringS2S.v2 | 25.33% | default: 25.33% |
| StackExchangeClustering.v2 | 44.13% | default: 44.13% |
| StackExchangeClusteringP2P.v2 | 33.07% | default: 33.07% |
| TwentyNewsgroupsClustering.v2 | 22.01% | default: 22.01% |
| Task | Average | Details |
|---|---|---|
| BIOSSES | 57.32% | default: 57.32% |
| SICK-R | 69.91% | default: 69.91% |
| STS12 | 66.88% | default: 66.88% |
| STS13 | 71.42% | default: 71.42% |
| STS14 | 68.52% | default: 68.52% |
| STS15 | 79.84% | default: 79.84% |
| STS17 | 53.52% | en-en: 82.46%, es-es: 78.22%, ko-ko: 66.78% |
| STS22.v2 | 40.57% | zh: 59.49%, es: 58.65%, fr: 57.4% |
| STSBenchmark | 74.69% | default: 74.69% |
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (12 layers, 384d)[0, 2, 4, 7, 9, 11] - 6 layers, evenly spaced (general-purpose)