Token-embedding pruning clusters semantically similar tokens in the embedding space (using DBSCAN) and merges each cluster into a single shared embedding, shrinking the vocabulary and reducing memory without retraining the transformer layers.

How to use

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jangedoo/multilingual-e5-small-ne-pruned", 
                            trust_remote_code=True)
embeddings = model.encode(["Hello world", "How are you?"])

Note: trust_remote_code=True is required because the model ships a small custom tokenizer class (pruned_tokenizer.py) that applies the id remapping after tokenization. No additional package installation is needed.

Pruning statistics

	Base	Pruned	Reduction
Vocab size	250,037	20,928	91.63%
Total parameters	117,653,760	29,675,904	74.78%
Embedding parameters	96,014,208	8,036,352	91.63%
Embedding size (MB)	366.3	30.7	335.6 MB saved

Evaluation

Dataset / Metric	Base	Pruned	Relative (base = 1.0)
stsb_ne / stsb_ne_pearson_cosine	0.7227	0.7205	0.9971
stsb_ne / stsb_ne_spearman_cosine	0.7182	0.7156	0.9964
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@1	0.1000	0.0600	0.6000
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@3	0.3000	0.3200	1.0667
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@5	0.4200	0.4000	0.9524
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@10	0.5400	0.5400	1.0000
nanobeir_ne / NanoClimateFEVER_cosine_precision@1	0.1000	0.0600	0.6000
nanobeir_ne / NanoClimateFEVER_cosine_precision@3	0.1000	0.1067	1.0667
nanobeir_ne / NanoClimateFEVER_cosine_precision@5	0.1000	0.0880	0.8800
nanobeir_ne / NanoClimateFEVER_cosine_precision@10	0.0700	0.0680	0.9714
nanobeir_ne / NanoClimateFEVER_cosine_recall@1	0.0300	0.0167	0.5556
nanobeir_ne / NanoClimateFEVER_cosine_recall@3	0.1400	0.1583	1.1310
nanobeir_ne / NanoClimateFEVER_cosine_recall@5	0.2073	0.1917	0.9244
nanobeir_ne / NanoClimateFEVER_cosine_recall@10	0.2747	0.2730	0.9939
nanobeir_ne / NanoClimateFEVER_cosine_ndcg@10	0.1836	0.1709	0.9307
nanobeir_ne / NanoClimateFEVER_cosine_mrr@10	0.2279	0.1978	0.8680
nanobeir_ne / NanoClimateFEVER_cosine_map@100	0.1241	0.1095	0.8820
nanobeir_ne / NanoDBPedia_cosine_accuracy@1	0.4000	0.4000	1.0000
nanobeir_ne / NanoDBPedia_cosine_accuracy@3	0.7600	0.6600	0.8684
nanobeir_ne / NanoDBPedia_cosine_accuracy@5	0.8000	0.7800	0.9750
nanobeir_ne / NanoDBPedia_cosine_accuracy@10	0.8200	0.8400	1.0244
nanobeir_ne / NanoDBPedia_cosine_precision@1	0.4000	0.4000	1.0000
nanobeir_ne / NanoDBPedia_cosine_precision@3	0.4133	0.3600	0.8710
nanobeir_ne / NanoDBPedia_cosine_precision@5	0.3680	0.3760	1.0217
nanobeir_ne / NanoDBPedia_cosine_precision@10	0.3260	0.3080	0.9448
nanobeir_ne / NanoDBPedia_cosine_recall@1	0.0736	0.0487	0.6611
nanobeir_ne / NanoDBPedia_cosine_recall@3	0.1475	0.1216	0.8243
nanobeir_ne / NanoDBPedia_cosine_recall@5	0.1746	0.1643	0.9407
nanobeir_ne / NanoDBPedia_cosine_recall@10	0.2453	0.2246	0.9153
nanobeir_ne / NanoDBPedia_cosine_ndcg@10	0.4156	0.3806	0.9157
nanobeir_ne / NanoDBPedia_cosine_mrr@10	0.5748	0.5501	0.9570
nanobeir_ne / NanoDBPedia_cosine_map@100	0.3034	0.2629	0.8665
nanobeir_ne / NanoFEVER_cosine_accuracy@1	0.3400	0.2600	0.7647
nanobeir_ne / NanoFEVER_cosine_accuracy@3	0.5800	0.5600	0.9655
nanobeir_ne / NanoFEVER_cosine_accuracy@5	0.6600	0.6000	0.9091
nanobeir_ne / NanoFEVER_cosine_accuracy@10	0.8000	0.6800	0.8500
nanobeir_ne / NanoFEVER_cosine_precision@1	0.3400	0.2600	0.7647
nanobeir_ne / NanoFEVER_cosine_precision@3	0.1933	0.1867	0.9655
nanobeir_ne / NanoFEVER_cosine_precision@5	0.1360	0.1240	0.9118
nanobeir_ne / NanoFEVER_cosine_precision@10	0.0820	0.0700	0.8537
nanobeir_ne / NanoFEVER_cosine_recall@1	0.3267	0.2467	0.7551
nanobeir_ne / NanoFEVER_cosine_recall@3	0.5567	0.5367	0.9641
nanobeir_ne / NanoFEVER_cosine_recall@5	0.6467	0.5867	0.9072
nanobeir_ne / NanoFEVER_cosine_recall@10	0.7767	0.6667	0.8584
nanobeir_ne / NanoFEVER_cosine_ndcg@10	0.5473	0.4623	0.8448
nanobeir_ne / NanoFEVER_cosine_mrr@10	0.4854	0.4062	0.8369
nanobeir_ne / NanoFEVER_cosine_map@100	0.4794	0.4061	0.8471
nanobeir_ne / NanoFiQA2018_cosine_accuracy@1	0.2600	0.2400	0.9231
nanobeir_ne / NanoFiQA2018_cosine_accuracy@3	0.4200	0.4000	0.9524
nanobeir_ne / NanoFiQA2018_cosine_accuracy@5	0.4600	0.4800	1.0435
nanobeir_ne / NanoFiQA2018_cosine_accuracy@10	0.5400	0.5200	0.9630
nanobeir_ne / NanoFiQA2018_cosine_precision@1	0.2600	0.2400	0.9231
nanobeir_ne / NanoFiQA2018_cosine_precision@3	0.1600	0.1600	1.0000
nanobeir_ne / NanoFiQA2018_cosine_precision@5	0.1240	0.1320	1.0645
nanobeir_ne / NanoFiQA2018_cosine_precision@10	0.0800	0.0740	0.9250
nanobeir_ne / NanoFiQA2018_cosine_recall@1	0.1287	0.1259	0.9778
nanobeir_ne / NanoFiQA2018_cosine_recall@3	0.2288	0.2437	1.0654
nanobeir_ne / NanoFiQA2018_cosine_recall@5	0.2893	0.3242	1.1208
nanobeir_ne / NanoFiQA2018_cosine_recall@10	0.3780	0.3518	0.9307
nanobeir_ne / NanoFiQA2018_cosine_ndcg@10	0.2912	0.2832	0.9728
nanobeir_ne / NanoFiQA2018_cosine_mrr@10	0.3572	0.3407	0.9538
nanobeir_ne / NanoFiQA2018_cosine_map@100	0.2275	0.2318	1.0186
nanobeir_ne / NanoHotpotQA_cosine_accuracy@1	0.7800	0.6200	0.7949
nanobeir_ne / NanoHotpotQA_cosine_accuracy@3	0.8400	0.7000	0.8333
nanobeir_ne / NanoHotpotQA_cosine_accuracy@5	0.8600	0.7200	0.8372
nanobeir_ne / NanoHotpotQA_cosine_accuracy@10	0.9000	0.8400	0.9333
nanobeir_ne / NanoHotpotQA_cosine_precision@1	0.7800	0.6200	0.7949
nanobeir_ne / NanoHotpotQA_cosine_precision@3	0.3800	0.3200	0.8421
nanobeir_ne / NanoHotpotQA_cosine_precision@5	0.2520	0.2160	0.8571
nanobeir_ne / NanoHotpotQA_cosine_precision@10	0.1380	0.1260	0.9130
nanobeir_ne / NanoHotpotQA_cosine_recall@1	0.3900	0.3100	0.7949
nanobeir_ne / NanoHotpotQA_cosine_recall@3	0.5700	0.4800	0.8421
nanobeir_ne / NanoHotpotQA_cosine_recall@5	0.6300	0.5400	0.8571
nanobeir_ne / NanoHotpotQA_cosine_recall@10	0.6900	0.6300	0.9130
nanobeir_ne / NanoHotpotQA_cosine_ndcg@10	0.6636	0.5743	0.8654
nanobeir_ne / NanoHotpotQA_cosine_mrr@10	0.8132	0.6824	0.8392
nanobeir_ne / NanoHotpotQA_cosine_map@100	0.5941	0.5047	0.8495
nanobeir_ne / NanoMSMARCO_cosine_accuracy@1	0.2600	0.2200	0.8462
nanobeir_ne / NanoMSMARCO_cosine_accuracy@3	0.5800	0.5000	0.8621
nanobeir_ne / NanoMSMARCO_cosine_accuracy@5	0.6600	0.6200	0.9394
nanobeir_ne / NanoMSMARCO_cosine_accuracy@10	0.7400	0.7000	0.9459
nanobeir_ne / NanoMSMARCO_cosine_precision@1	0.2600	0.2200	0.8462
nanobeir_ne / NanoMSMARCO_cosine_precision@3	0.1933	0.1667	0.8621
nanobeir_ne / NanoMSMARCO_cosine_precision@5	0.1320	0.1240	0.9394
nanobeir_ne / NanoMSMARCO_cosine_precision@10	0.0740	0.0700	0.9459
nanobeir_ne / NanoMSMARCO_cosine_recall@1	0.2600	0.2200	0.8462
nanobeir_ne / NanoMSMARCO_cosine_recall@3	0.5800	0.5000	0.8621
nanobeir_ne / NanoMSMARCO_cosine_recall@5	0.6600	0.6200	0.9394
nanobeir_ne / NanoMSMARCO_cosine_recall@10	0.7400	0.7000	0.9459
nanobeir_ne / NanoMSMARCO_cosine_ndcg@10	0.4955	0.4545	0.9172
nanobeir_ne / NanoMSMARCO_cosine_mrr@10	0.4174	0.3764	0.9018
nanobeir_ne / NanoMSMARCO_cosine_map@100	0.4252	0.3840	0.9030
nanobeir_ne / NanoNFCorpus_cosine_accuracy@1	0.2800	0.2600	0.9286
nanobeir_ne / NanoNFCorpus_cosine_accuracy@3	0.4400	0.3800	0.8636
nanobeir_ne / NanoNFCorpus_cosine_accuracy@5	0.4400	0.4400	1.0000
nanobeir_ne / NanoNFCorpus_cosine_accuracy@10	0.4400	0.4800	1.0909
nanobeir_ne / NanoNFCorpus_cosine_precision@1	0.2800	0.2600	0.9286
nanobeir_ne / NanoNFCorpus_cosine_precision@3	0.2600	0.2467	0.9487
nanobeir_ne / NanoNFCorpus_cosine_precision@5	0.2120	0.2240	1.0566
nanobeir_ne / NanoNFCorpus_cosine_precision@10	0.1600	0.1660	1.0375
nanobeir_ne / NanoNFCorpus_cosine_recall@1	0.0084	0.0072	0.8593
nanobeir_ne / NanoNFCorpus_cosine_recall@3	0.0413	0.0316	0.7637
nanobeir_ne / NanoNFCorpus_cosine_recall@5	0.0486	0.0530	1.0902
nanobeir_ne / NanoNFCorpus_cosine_recall@10	0.0617	0.0862	1.3967
nanobeir_ne / NanoNFCorpus_cosine_ndcg@10	0.1975	0.2019	1.0223
nanobeir_ne / NanoNFCorpus_cosine_mrr@10	0.3500	0.3306	0.9444
nanobeir_ne / NanoNFCorpus_cosine_map@100	0.0701	0.0685	0.9775
nanobeir_ne / NanoNQ_cosine_accuracy@1	0.2000	0.1800	0.9000
nanobeir_ne / NanoNQ_cosine_accuracy@3	0.3400	0.2800	0.8235
nanobeir_ne / NanoNQ_cosine_accuracy@5	0.3400	0.3000	0.8824
nanobeir_ne / NanoNQ_cosine_accuracy@10	0.4400	0.4400	1.0000
nanobeir_ne / NanoNQ_cosine_precision@1	0.2000	0.1800	0.9000
nanobeir_ne / NanoNQ_cosine_precision@3	0.1133	0.0933	0.8235
nanobeir_ne / NanoNQ_cosine_precision@5	0.0680	0.0600	0.8824
nanobeir_ne / NanoNQ_cosine_precision@10	0.0440	0.0440	1.0000
nanobeir_ne / NanoNQ_cosine_recall@1	0.1800	0.1700	0.9444
nanobeir_ne / NanoNQ_cosine_recall@3	0.3100	0.2600	0.8387
nanobeir_ne / NanoNQ_cosine_recall@5	0.3100	0.2800	0.9032
nanobeir_ne / NanoNQ_cosine_recall@10	0.4100	0.4100	1.0000
nanobeir_ne / NanoNQ_cosine_ndcg@10	0.2951	0.2770	0.9385
nanobeir_ne / NanoNQ_cosine_mrr@10	0.2767	0.2480	0.8963
nanobeir_ne / NanoNQ_cosine_map@100	0.2685	0.2469	0.9197
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@1	0.8200	0.8000	0.9756
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@3	0.9000	0.9000	1.0000
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@5	0.9200	0.9200	1.0000
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@10	0.9800	0.9800	1.0000
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@1	0.8200	0.8000	0.9756
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@3	0.3533	0.3533	1.0000
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@5	0.2360	0.2320	0.9831
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@10	0.1320	0.1300	0.9848
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@1	0.7240	0.7040	0.9724
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@3	0.8380	0.8380	1.0000
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@5	0.8860	0.8793	0.9925
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@10	0.9660	0.9593	0.9931
nanobeir_ne / NanoQuoraRetrieval_cosine_ndcg@10	0.8797	0.8678	0.9864
nanobeir_ne / NanoQuoraRetrieval_cosine_mrr@10	0.8707	0.8574	0.9847
nanobeir_ne / NanoQuoraRetrieval_cosine_map@100	0.8452	0.8319	0.9842
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@1	0.2000	0.1800	0.9000
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@3	0.3600	0.3200	0.8889
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@5	0.4600	0.4800	1.0435
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@10	0.5800	0.6200	1.0690
nanobeir_ne / NanoSCIDOCS_cosine_precision@1	0.2000	0.1800	0.9000
nanobeir_ne / NanoSCIDOCS_cosine_precision@3	0.1733	0.1467	0.8462
nanobeir_ne / NanoSCIDOCS_cosine_precision@5	0.1480	0.1360	0.9189
nanobeir_ne / NanoSCIDOCS_cosine_precision@10	0.0980	0.0960	0.9796
nanobeir_ne / NanoSCIDOCS_cosine_recall@1	0.0420	0.0370	0.8810
nanobeir_ne / NanoSCIDOCS_cosine_recall@3	0.1080	0.0910	0.8426
nanobeir_ne / NanoSCIDOCS_cosine_recall@5	0.1557	0.1437	0.9229
nanobeir_ne / NanoSCIDOCS_cosine_recall@10	0.2047	0.2007	0.9805
nanobeir_ne / NanoSCIDOCS_cosine_ndcg@10	0.1883	0.1781	0.9459
nanobeir_ne / NanoSCIDOCS_cosine_mrr@10	0.3002	0.2899	0.9655
nanobeir_ne / NanoSCIDOCS_cosine_map@100	0.1343	0.1191	0.8868
nanobeir_ne / NanoArguAna_cosine_accuracy@1	0.1200	0.1200	1.0000
nanobeir_ne / NanoArguAna_cosine_accuracy@3	0.5200	0.4800	0.9231
nanobeir_ne / NanoArguAna_cosine_accuracy@5	0.5800	0.6000	1.0345
nanobeir_ne / NanoArguAna_cosine_accuracy@10	0.7400	0.7000	0.9459
nanobeir_ne / NanoArguAna_cosine_precision@1	0.1200	0.1200	1.0000
nanobeir_ne / NanoArguAna_cosine_precision@3	0.1733	0.1600	0.9231
nanobeir_ne / NanoArguAna_cosine_precision@5	0.1160	0.1200	1.0345
nanobeir_ne / NanoArguAna_cosine_precision@10	0.0740	0.0700	0.9459
nanobeir_ne / NanoArguAna_cosine_recall@1	0.1200	0.1200	1.0000
nanobeir_ne / NanoArguAna_cosine_recall@3	0.5200	0.4800	0.9231
nanobeir_ne / NanoArguAna_cosine_recall@5	0.5800	0.6000	1.0345
nanobeir_ne / NanoArguAna_cosine_recall@10	0.7400	0.7000	0.9459
nanobeir_ne / NanoArguAna_cosine_ndcg@10	0.4276	0.4142	0.9688
nanobeir_ne / NanoArguAna_cosine_mrr@10	0.3276	0.3220	0.9831
nanobeir_ne / NanoArguAna_cosine_map@100	0.3367	0.3322	0.9867
nanobeir_ne / NanoSciFact_cosine_accuracy@1	0.3200	0.3400	1.0625
nanobeir_ne / NanoSciFact_cosine_accuracy@3	0.4800	0.4200	0.8750
nanobeir_ne / NanoSciFact_cosine_accuracy@5	0.5400	0.5200	0.9630
nanobeir_ne / NanoSciFact_cosine_accuracy@10	0.6600	0.5800	0.8788
nanobeir_ne / NanoSciFact_cosine_precision@1	0.3200	0.3400	1.0625
nanobeir_ne / NanoSciFact_cosine_precision@3	0.1667	0.1400	0.8400
nanobeir_ne / NanoSciFact_cosine_precision@5	0.1120	0.1080	0.9643
nanobeir_ne / NanoSciFact_cosine_precision@10	0.0700	0.0640	0.9143
nanobeir_ne / NanoSciFact_cosine_recall@1	0.3050	0.3400	1.1148
nanobeir_ne / NanoSciFact_cosine_recall@3	0.4600	0.3950	0.8587
nanobeir_ne / NanoSciFact_cosine_recall@5	0.5100	0.4900	0.9608
nanobeir_ne / NanoSciFact_cosine_recall@10	0.6250	0.5650	0.9040
nanobeir_ne / NanoSciFact_cosine_ndcg@10	0.4596	0.4435	0.9650
nanobeir_ne / NanoSciFact_cosine_mrr@10	0.4155	0.4092	0.9847
nanobeir_ne / NanoSciFact_cosine_map@100	0.4093	0.4098	1.0012
nanobeir_ne / NanoTouche2020_cosine_accuracy@1	0.3469	0.3469	1.0000
nanobeir_ne / NanoTouche2020_cosine_accuracy@3	0.5510	0.5714	1.0370
nanobeir_ne / NanoTouche2020_cosine_accuracy@5	0.6735	0.6735	1.0000
nanobeir_ne / NanoTouche2020_cosine_accuracy@10	0.8571	0.8776	1.0238
nanobeir_ne / NanoTouche2020_cosine_precision@1	0.3469	0.3469	1.0000
nanobeir_ne / NanoTouche2020_cosine_precision@3	0.3333	0.3401	1.0204
nanobeir_ne / NanoTouche2020_cosine_precision@5	0.3224	0.3265	1.0127
nanobeir_ne / NanoTouche2020_cosine_precision@10	0.2939	0.2857	0.9722
nanobeir_ne / NanoTouche2020_cosine_recall@1	0.0216	0.0216	1.0000
nanobeir_ne / NanoTouche2020_cosine_recall@3	0.0678	0.0687	1.0131
nanobeir_ne / NanoTouche2020_cosine_recall@5	0.1083	0.1075	0.9923
nanobeir_ne / NanoTouche2020_cosine_recall@10	0.1892	0.1824	0.9637
nanobeir_ne / NanoTouche2020_cosine_ndcg@10	0.3143	0.3079	0.9798
nanobeir_ne / NanoTouche2020_cosine_mrr@10	0.4881	0.4885	1.0009
nanobeir_ne / NanoTouche2020_cosine_map@100	0.2267	0.2246	0.9906
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@1	0.3405	0.3098	0.9096
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@3	0.5439	0.4993	0.9180
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@5	0.6010	0.5795	0.9642
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@10	0.6952	0.6767	0.9735
nanobeir_ne / NanoBEIR_mean_cosine_precision@1	0.3405	0.3098	0.9096
nanobeir_ne / NanoBEIR_mean_cosine_precision@3	0.2318	0.2139	0.9226
nanobeir_ne / NanoBEIR_mean_cosine_precision@5	0.1790	0.1743	0.9742
nanobeir_ne / NanoBEIR_mean_cosine_precision@10	0.1263	0.1209	0.9573
nanobeir_ne / NanoBEIR_mean_cosine_recall@1	0.2008	0.1821	0.9072
nanobeir_ne / NanoBEIR_mean_cosine_recall@3	0.3514	0.3234	0.9204
nanobeir_ne / NanoBEIR_mean_cosine_recall@5	0.4005	0.3831	0.9566
nanobeir_ne / NanoBEIR_mean_cosine_recall@10	0.4847	0.4577	0.9442
nanobeir_ne / NanoBEIR_mean_cosine_ndcg@10	0.4122	0.3859	0.9361
nanobeir_ne / NanoBEIR_mean_cosine_mrr@10	0.4542	0.4230	0.9313
nanobeir_ne / NanoBEIR_mean_cosine_map@100	0.3419	0.3178	0.9296

Citation

If you use this model or the pruning approach, please cite:

bibtex

@misc{subedi2025tokenpruning,
  author = {Sanjaya Subedi},
  title  = {Token Embedding Pruning for Sentence Transformers},
  year   = {2026},
  note   = {Available at: https://sanjayasubedi.com.np/deeplearning/shrinking-embedding-models-by-pruning-vocabulary/}
}