Token-embedding pruning clusters semantically similar tokens in the embedding space (using DBSCAN) and merges each cluster into a single shared embedding, shrinking the vocabulary and reducing memory without retraining the transformer layers.

How to use

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("./pruned/all-MiniLM-L6-v2-pruned", trust_remote_code=True)
embeddings = model.encode(["Hello world", "How are you?"])

Note: trust_remote_code=True is required because the model ships a small custom tokenizer class (pruned_tokenizer.py) that applies the id remapping after tokenization. No additional package installation is needed.

Pruning statistics

	Base	Pruned	Reduction
Vocab size	30,522	20,808	31.83%
Total parameters	22,713,216	18,983,040	16.42%
Embedding parameters	11,720,448	7,990,272	31.83%
Embedding size (MB)	44.7	30.5	14.2 MB saved

Evaluation

Dataset / Metric	Base	Pruned	Relative (base = 1.0)
stsb / stsb_pearson_cosine	0.8274	0.8183	0.9890
stsb / stsb_spearman_cosine	0.8203	0.8095	0.9868
nanobeir / NanoClimateFEVER_cosine_accuracy@1	0.2800	0.2200	0.7857
nanobeir / NanoClimateFEVER_cosine_accuracy@3	0.4400	0.4800	1.0909
nanobeir / NanoClimateFEVER_cosine_accuracy@5	0.5400	0.5800	1.0741
nanobeir / NanoClimateFEVER_cosine_accuracy@10	0.7200	0.7000	0.9722
nanobeir / NanoClimateFEVER_cosine_precision@1	0.2800	0.2200	0.7857
nanobeir / NanoClimateFEVER_cosine_precision@3	0.1533	0.1667	1.0870
nanobeir / NanoClimateFEVER_cosine_precision@5	0.1240	0.1360	1.0968
nanobeir / NanoClimateFEVER_cosine_precision@10	0.0900	0.0900	1.0000
nanobeir / NanoClimateFEVER_cosine_recall@1	0.1450	0.1000	0.6897
nanobeir / NanoClimateFEVER_cosine_recall@3	0.2050	0.2133	1.0407
nanobeir / NanoClimateFEVER_cosine_recall@5	0.2640	0.2990	1.1326
nanobeir / NanoClimateFEVER_cosine_recall@10	0.3620	0.3663	1.0120
nanobeir / NanoClimateFEVER_cosine_ndcg@10	0.2958	0.2794	0.9449
nanobeir / NanoClimateFEVER_cosine_mrr@10	0.3997	0.3679	0.9205
nanobeir / NanoClimateFEVER_cosine_map@100	0.2326	0.2087	0.8975
nanobeir / NanoDBPedia_cosine_accuracy@1	0.6800	0.6200	0.9118
nanobeir / NanoDBPedia_cosine_accuracy@3	0.8600	0.8200	0.9535
nanobeir / NanoDBPedia_cosine_accuracy@5	0.9200	0.8600	0.9348
nanobeir / NanoDBPedia_cosine_accuracy@10	0.9600	0.9200	0.9583
nanobeir / NanoDBPedia_cosine_precision@1	0.6800	0.6200	0.9118
nanobeir / NanoDBPedia_cosine_precision@3	0.5600	0.5200	0.9286
nanobeir / NanoDBPedia_cosine_precision@5	0.5120	0.4600	0.8984
nanobeir / NanoDBPedia_cosine_precision@10	0.4380	0.4180	0.9543
nanobeir / NanoDBPedia_cosine_recall@1	0.0760	0.0650	0.8553
nanobeir / NanoDBPedia_cosine_recall@3	0.1439	0.1321	0.9180
nanobeir / NanoDBPedia_cosine_recall@5	0.2068	0.1790	0.8654
nanobeir / NanoDBPedia_cosine_recall@10	0.3200	0.2839	0.8872
nanobeir / NanoDBPedia_cosine_ndcg@10	0.5501	0.5102	0.9274
nanobeir / NanoDBPedia_cosine_mrr@10	0.7855	0.7335	0.9338
nanobeir / NanoDBPedia_cosine_map@100	0.3948	0.3699	0.9371
nanobeir / NanoFEVER_cosine_accuracy@1	0.6800	0.5400	0.7941
nanobeir / NanoFEVER_cosine_accuracy@3	0.8600	0.8000	0.9302
nanobeir / NanoFEVER_cosine_accuracy@5	0.9200	0.9200	1.0000
nanobeir / NanoFEVER_cosine_accuracy@10	0.9600	0.9600	1.0000
nanobeir / NanoFEVER_cosine_precision@1	0.6800	0.5400	0.7941
nanobeir / NanoFEVER_cosine_precision@3	0.2933	0.2733	0.9318
nanobeir / NanoFEVER_cosine_precision@5	0.1920	0.1920	1.0000
nanobeir / NanoFEVER_cosine_precision@10	0.1020	0.1020	1.0000
nanobeir / NanoFEVER_cosine_recall@1	0.6267	0.5167	0.8245
nanobeir / NanoFEVER_cosine_recall@3	0.8133	0.7633	0.9385
nanobeir / NanoFEVER_cosine_recall@5	0.8833	0.8733	0.9887
nanobeir / NanoFEVER_cosine_recall@10	0.9233	0.9233	1.0000
nanobeir / NanoFEVER_cosine_ndcg@10	0.7933	0.7317	0.9223
nanobeir / NanoFEVER_cosine_mrr@10	0.7781	0.6846	0.8798
nanobeir / NanoFEVER_cosine_map@100	0.7407	0.6631	0.8953
nanobeir / NanoFiQA2018_cosine_accuracy@1	0.4600	0.4600	1.0000
nanobeir / NanoFiQA2018_cosine_accuracy@3	0.6400	0.6600	1.0312
nanobeir / NanoFiQA2018_cosine_accuracy@5	0.7000	0.6800	0.9714
nanobeir / NanoFiQA2018_cosine_accuracy@10	0.7200	0.7600	1.0556
nanobeir / NanoFiQA2018_cosine_precision@1	0.4600	0.4600	1.0000
nanobeir / NanoFiQA2018_cosine_precision@3	0.2867	0.2933	1.0233
nanobeir / NanoFiQA2018_cosine_precision@5	0.2240	0.2160	0.9643
nanobeir / NanoFiQA2018_cosine_precision@10	0.1300	0.1240	0.9538
nanobeir / NanoFiQA2018_cosine_recall@1	0.2392	0.2709	1.1324
nanobeir / NanoFiQA2018_cosine_recall@3	0.4251	0.4355	1.0244
nanobeir / NanoFiQA2018_cosine_recall@5	0.5100	0.5046	0.9895
nanobeir / NanoFiQA2018_cosine_recall@10	0.5660	0.5751	1.0161
nanobeir / NanoFiQA2018_cosine_ndcg@10	0.4775	0.4907	1.0277
nanobeir / NanoFiQA2018_cosine_mrr@10	0.5476	0.5602	1.0231
nanobeir / NanoFiQA2018_cosine_map@100	0.4125	0.4275	1.0363
nanobeir / NanoHotpotQA_cosine_accuracy@1	0.6400	0.5600	0.8750
nanobeir / NanoHotpotQA_cosine_accuracy@3	0.8200	0.8000	0.9756
nanobeir / NanoHotpotQA_cosine_accuracy@5	0.8400	0.8200	0.9762
nanobeir / NanoHotpotQA_cosine_accuracy@10	0.8800	0.8400	0.9545
nanobeir / NanoHotpotQA_cosine_precision@1	0.6400	0.5600	0.8750
nanobeir / NanoHotpotQA_cosine_precision@3	0.3533	0.3667	1.0377
nanobeir / NanoHotpotQA_cosine_precision@5	0.2360	0.2240	0.9492
nanobeir / NanoHotpotQA_cosine_precision@10	0.1280	0.1240	0.9687
nanobeir / NanoHotpotQA_cosine_recall@1	0.3200	0.2800	0.8750
nanobeir / NanoHotpotQA_cosine_recall@3	0.5300	0.5500	1.0377
nanobeir / NanoHotpotQA_cosine_recall@5	0.5900	0.5600	0.9492
nanobeir / NanoHotpotQA_cosine_recall@10	0.6400	0.6200	0.9688
nanobeir / NanoHotpotQA_cosine_ndcg@10	0.5960	0.5644	0.9471
nanobeir / NanoHotpotQA_cosine_mrr@10	0.7239	0.6683	0.9233
nanobeir / NanoHotpotQA_cosine_map@100	0.5262	0.5004	0.9509
nanobeir / NanoMSMARCO_cosine_accuracy@1	0.3600	0.3000	0.8333
nanobeir / NanoMSMARCO_cosine_accuracy@3	0.5200	0.5200	1.0000
nanobeir / NanoMSMARCO_cosine_accuracy@5	0.5800	0.6600	1.1379
nanobeir / NanoMSMARCO_cosine_accuracy@10	0.8000	0.7200	0.9000
nanobeir / NanoMSMARCO_cosine_precision@1	0.3600	0.3000	0.8333
nanobeir / NanoMSMARCO_cosine_precision@3	0.1733	0.1733	1.0000
nanobeir / NanoMSMARCO_cosine_precision@5	0.1160	0.1320	1.1379
nanobeir / NanoMSMARCO_cosine_precision@10	0.0800	0.0720	0.9000
nanobeir / NanoMSMARCO_cosine_recall@1	0.3600	0.3000	0.8333
nanobeir / NanoMSMARCO_cosine_recall@3	0.5200	0.5200	1.0000
nanobeir / NanoMSMARCO_cosine_recall@5	0.5800	0.6600	1.1379
nanobeir / NanoMSMARCO_cosine_recall@10	0.8000	0.7200	0.9000
nanobeir / NanoMSMARCO_cosine_ndcg@10	0.5540	0.5126	0.9253
nanobeir / NanoMSMARCO_cosine_mrr@10	0.4796	0.4450	0.9279
nanobeir / NanoMSMARCO_cosine_map@100	0.4908	0.4610	0.9394
nanobeir / NanoNFCorpus_cosine_accuracy@1	0.4200	0.3800	0.9048
nanobeir / NanoNFCorpus_cosine_accuracy@3	0.5600	0.5200	0.9286
nanobeir / NanoNFCorpus_cosine_accuracy@5	0.6000	0.6000	1.0000
nanobeir / NanoNFCorpus_cosine_accuracy@10	0.7000	0.7200	1.0286
nanobeir / NanoNFCorpus_cosine_precision@1	0.4200	0.3800	0.9048
nanobeir / NanoNFCorpus_cosine_precision@3	0.3467	0.3533	1.0192
nanobeir / NanoNFCorpus_cosine_precision@5	0.3280	0.3160	0.9634
nanobeir / NanoNFCorpus_cosine_precision@10	0.2860	0.2800	0.9790
nanobeir / NanoNFCorpus_cosine_recall@1	0.0339	0.0139	0.4099
nanobeir / NanoNFCorpus_cosine_recall@3	0.0631	0.0478	0.7575
nanobeir / NanoNFCorpus_cosine_recall@5	0.0819	0.0699	0.8536
nanobeir / NanoNFCorpus_cosine_recall@10	0.1348	0.1314	0.9749
nanobeir / NanoNFCorpus_cosine_ndcg@10	0.3322	0.3145	0.9468
nanobeir / NanoNFCorpus_cosine_mrr@10	0.4983	0.4788	0.9608
nanobeir / NanoNFCorpus_cosine_map@100	0.1398	0.1216	0.8699
nanobeir / NanoNQ_cosine_accuracy@1	0.4400	0.4000	0.9091
nanobeir / NanoNQ_cosine_accuracy@3	0.6400	0.6200	0.9688
nanobeir / NanoNQ_cosine_accuracy@5	0.6600	0.6800	1.0303
nanobeir / NanoNQ_cosine_accuracy@10	0.7600	0.7200	0.9474
nanobeir / NanoNQ_cosine_precision@1	0.4400	0.4000	0.9091
nanobeir / NanoNQ_cosine_precision@3	0.2200	0.2133	0.9697
nanobeir / NanoNQ_cosine_precision@5	0.1400	0.1440	1.0286
nanobeir / NanoNQ_cosine_precision@10	0.0820	0.0780	0.9512
nanobeir / NanoNQ_cosine_recall@1	0.4200	0.3800	0.9048
nanobeir / NanoNQ_cosine_recall@3	0.6200	0.5900	0.9516
nanobeir / NanoNQ_cosine_recall@5	0.6400	0.6600	1.0312
nanobeir / NanoNQ_cosine_recall@10	0.7500	0.7100	0.9467
nanobeir / NanoNQ_cosine_ndcg@10	0.5904	0.5533	0.9372
nanobeir / NanoNQ_cosine_mrr@10	0.5456	0.5091	0.9330
nanobeir / NanoNQ_cosine_map@100	0.5437	0.5080	0.9342
nanobeir / NanoQuoraRetrieval_cosine_accuracy@1	0.8800	0.8600	0.9773
nanobeir / NanoQuoraRetrieval_cosine_accuracy@3	0.9600	0.9800	1.0208
nanobeir / NanoQuoraRetrieval_cosine_accuracy@5	1.0000	0.9800	0.9800
nanobeir / NanoQuoraRetrieval_cosine_accuracy@10	1.0000	1.0000	1.0000
nanobeir / NanoQuoraRetrieval_cosine_precision@1	0.8800	0.8600	0.9773
nanobeir / NanoQuoraRetrieval_cosine_precision@3	0.3933	0.4000	1.0169
nanobeir / NanoQuoraRetrieval_cosine_precision@5	0.2560	0.2560	1.0000
nanobeir / NanoQuoraRetrieval_cosine_precision@10	0.1360	0.1340	0.9853
nanobeir / NanoQuoraRetrieval_cosine_recall@1	0.7840	0.7473	0.9532
nanobeir / NanoQuoraRetrieval_cosine_recall@3	0.9187	0.9387	1.0218
nanobeir / NanoQuoraRetrieval_cosine_recall@5	0.9760	0.9600	0.9836
nanobeir / NanoQuoraRetrieval_cosine_recall@10	0.9933	0.9900	0.9966
nanobeir / NanoQuoraRetrieval_cosine_ndcg@10	0.9368	0.9241	0.9864
nanobeir / NanoQuoraRetrieval_cosine_mrr@10	0.9247	0.9156	0.9901
nanobeir / NanoQuoraRetrieval_cosine_map@100	0.9136	0.8978	0.9827
nanobeir / NanoSCIDOCS_cosine_accuracy@1	0.5200	0.4800	0.9231
nanobeir / NanoSCIDOCS_cosine_accuracy@3	0.6800	0.7400	1.0882
nanobeir / NanoSCIDOCS_cosine_accuracy@5	0.8200	0.8000	0.9756
nanobeir / NanoSCIDOCS_cosine_accuracy@10	0.9200	0.9000	0.9783
nanobeir / NanoSCIDOCS_cosine_precision@1	0.5200	0.4800	0.9231
nanobeir / NanoSCIDOCS_cosine_precision@3	0.3933	0.3733	0.9492
nanobeir / NanoSCIDOCS_cosine_precision@5	0.3360	0.3080	0.9167
nanobeir / NanoSCIDOCS_cosine_precision@10	0.2160	0.2120	0.9815
nanobeir / NanoSCIDOCS_cosine_recall@1	0.1097	0.1007	0.9179
nanobeir / NanoSCIDOCS_cosine_recall@3	0.2447	0.2317	0.9469
nanobeir / NanoSCIDOCS_cosine_recall@5	0.3457	0.3167	0.9161
nanobeir / NanoSCIDOCS_cosine_recall@10	0.4427	0.4357	0.9842
nanobeir / NanoSCIDOCS_cosine_ndcg@10	0.4328	0.4143	0.9573
nanobeir / NanoSCIDOCS_cosine_mrr@10	0.6317	0.6142	0.9723
nanobeir / NanoSCIDOCS_cosine_map@100	0.3500	0.3276	0.9360
nanobeir / NanoArguAna_cosine_accuracy@1	0.2000	0.1800	0.9000
nanobeir / NanoArguAna_cosine_accuracy@3	0.5600	0.5800	1.0357
nanobeir / NanoArguAna_cosine_accuracy@5	0.7600	0.7000	0.9211
nanobeir / NanoArguAna_cosine_accuracy@10	0.9200	0.9000	0.9783
nanobeir / NanoArguAna_cosine_precision@1	0.2000	0.1800	0.9000
nanobeir / NanoArguAna_cosine_precision@3	0.1867	0.1933	1.0357
nanobeir / NanoArguAna_cosine_precision@5	0.1520	0.1400	0.9211
nanobeir / NanoArguAna_cosine_precision@10	0.0920	0.0900	0.9783
nanobeir / NanoArguAna_cosine_recall@1	0.2000	0.1800	0.9000
nanobeir / NanoArguAna_cosine_recall@3	0.5600	0.5800	1.0357
nanobeir / NanoArguAna_cosine_recall@5	0.7600	0.7000	0.9211
nanobeir / NanoArguAna_cosine_recall@10	0.9200	0.9000	0.9783
nanobeir / NanoArguAna_cosine_ndcg@10	0.5525	0.5323	0.9633
nanobeir / NanoArguAna_cosine_mrr@10	0.4356	0.4158	0.9544
nanobeir / NanoArguAna_cosine_map@100	0.4386	0.4195	0.9565
nanobeir / NanoSciFact_cosine_accuracy@1	0.6000	0.5600	0.9333
nanobeir / NanoSciFact_cosine_accuracy@3	0.7200	0.7000	0.9722
nanobeir / NanoSciFact_cosine_accuracy@5	0.8000	0.8000	1.0000
nanobeir / NanoSciFact_cosine_accuracy@10	0.8800	0.8800	1.0000
nanobeir / NanoSciFact_cosine_precision@1	0.6000	0.5600	0.9333
nanobeir / NanoSciFact_cosine_precision@3	0.2533	0.2467	0.9737
nanobeir / NanoSciFact_cosine_precision@5	0.1800	0.1720	0.9556
nanobeir / NanoSciFact_cosine_precision@10	0.0980	0.0980	1.0000
nanobeir / NanoSciFact_cosine_recall@1	0.5800	0.5400	0.9310
nanobeir / NanoSciFact_cosine_recall@3	0.7000	0.6800	0.9714
nanobeir / NanoSciFact_cosine_recall@5	0.8000	0.7850	0.9812
nanobeir / NanoSciFact_cosine_recall@10	0.8700	0.8700	1.0000
nanobeir / NanoSciFact_cosine_ndcg@10	0.7265	0.7096	0.9767
nanobeir / NanoSciFact_cosine_mrr@10	0.6841	0.6615	0.9670
nanobeir / NanoSciFact_cosine_map@100	0.6810	0.6577	0.9658
nanobeir / NanoTouche2020_cosine_accuracy@1	0.5102	0.4694	0.9200
nanobeir / NanoTouche2020_cosine_accuracy@3	0.8367	0.8367	1.0000
nanobeir / NanoTouche2020_cosine_accuracy@5	0.9184	0.8571	0.9333
nanobeir / NanoTouche2020_cosine_accuracy@10	0.9388	0.9796	1.0435
nanobeir / NanoTouche2020_cosine_precision@1	0.5102	0.4694	0.9200
nanobeir / NanoTouche2020_cosine_precision@3	0.5374	0.5306	0.9873
nanobeir / NanoTouche2020_cosine_precision@5	0.5061	0.4776	0.9435
nanobeir / NanoTouche2020_cosine_precision@10	0.4327	0.4102	0.9481
nanobeir / NanoTouche2020_cosine_recall@1	0.0355	0.0318	0.8966
nanobeir / NanoTouche2020_cosine_recall@3	0.1119	0.1069	0.9553
nanobeir / NanoTouche2020_cosine_recall@5	0.1674	0.1632	0.9752
nanobeir / NanoTouche2020_cosine_recall@10	0.2819	0.2724	0.9663
nanobeir / NanoTouche2020_cosine_ndcg@10	0.4748	0.4506	0.9490
nanobeir / NanoTouche2020_cosine_mrr@10	0.6714	0.6479	0.9649
nanobeir / NanoTouche2020_cosine_map@100	0.3438	0.3306	0.9614
nanobeir / NanoBEIR_mean_cosine_accuracy@1	0.5131	0.4638	0.9039
nanobeir / NanoBEIR_mean_cosine_accuracy@3	0.6997	0.6967	0.9956
nanobeir / NanoBEIR_mean_cosine_accuracy@5	0.7737	0.7644	0.9879
nanobeir / NanoBEIR_mean_cosine_accuracy@10	0.8584	0.8461	0.9857
nanobeir / NanoBEIR_mean_cosine_precision@1	0.5131	0.4638	0.9039
nanobeir / NanoBEIR_mean_cosine_precision@3	0.3193	0.3157	0.9887
nanobeir / NanoBEIR_mean_cosine_precision@5	0.2540	0.2441	0.9611
nanobeir / NanoBEIR_mean_cosine_precision@10	0.1777	0.1717	0.9660
nanobeir / NanoBEIR_mean_cosine_recall@1	0.3023	0.2713	0.8973
nanobeir / NanoBEIR_mean_cosine_recall@3	0.4504	0.4453	0.9887
nanobeir / NanoBEIR_mean_cosine_recall@5	0.5235	0.5177	0.9891
nanobeir / NanoBEIR_mean_cosine_recall@10	0.6157	0.5999	0.9743
nanobeir / NanoBEIR_mean_cosine_ndcg@10	0.5625	0.5375	0.9556
nanobeir / NanoBEIR_mean_cosine_mrr@10	0.6235	0.5925	0.9502
nanobeir / NanoBEIR_mean_cosine_map@100	0.4775	0.4533	0.9493

Citation

If you use this model or the pruning approach, please cite:

bibtex

@misc{subedi2025tokenpruning,
  author = {Sanjaya Subedi},
  title  = {Token Embedding Pruning for Sentence Transformers},
  year   = {2026},
  note   = {Available at: [link to be added upon publication]}
}

jangedoo/all-MiniLM-L6-v2-pruned

all-MiniLM-L6-v2-pruned

How to use

Pruning statistics

Evaluation

Citation

jangedoo/
all-MiniLM-L6-v2-pruned