Token-embedding pruning clusters semantically similar tokens in the embedding space (using DBSCAN) and merges each cluster into a single shared embedding, shrinking the vocabulary and reducing memory without retraining the transformer layers.

How to use

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jangedoo/embeddinggemma-300m-pruned", trust_remote_code=True)
embeddings = model.encode(["Hello world", "How are you?"])

Note: trust_remote_code=True is required because the model ships a small custom tokenizer class (pruned_tokenizer.py) that applies the id remapping after tokenization. No additional package installation is needed.

Pruning statistics

	Base	Pruned	Reduction
Vocab size	262,144	188,191	28.21%
Total parameters	307,581,696	250,785,792	18.47%
Embedding parameters	201,326,592	144,530,688	28.21%
Embedding size (MB)	768.0	551.3	216.7 MB saved

Evaluation

Dataset / Metric	Base	Pruned	Relative (base = 1.0)
stsb / stsb_pearson_cosine	0.8446	0.8113	0.9606
stsb / stsb_spearman_cosine	0.8485	0.8074	0.9515
nanobeir / NanoClimateFEVER_cosine_accuracy@1	0.3000	0.2800	0.9333
nanobeir / NanoClimateFEVER_cosine_accuracy@3	0.4400	0.4200	0.9545
nanobeir / NanoClimateFEVER_cosine_accuracy@5	0.6200	0.5600	0.9032
nanobeir / NanoClimateFEVER_cosine_accuracy@10	0.8000	0.7200	0.9000
nanobeir / NanoClimateFEVER_cosine_precision@1	0.3000	0.2800	0.9333
nanobeir / NanoClimateFEVER_cosine_precision@3	0.1733	0.1667	0.9615
nanobeir / NanoClimateFEVER_cosine_precision@5	0.1440	0.1400	0.9722
nanobeir / NanoClimateFEVER_cosine_precision@10	0.1100	0.0980	0.8909
nanobeir / NanoClimateFEVER_cosine_recall@1	0.1383	0.1033	0.7470
nanobeir / NanoClimateFEVER_cosine_recall@3	0.2367	0.2067	0.8732
nanobeir / NanoClimateFEVER_cosine_recall@5	0.3173	0.2807	0.8845
nanobeir / NanoClimateFEVER_cosine_recall@10	0.4413	0.3713	0.8414
nanobeir / NanoClimateFEVER_cosine_ndcg@10	0.3341	0.2909	0.8705
nanobeir / NanoClimateFEVER_cosine_mrr@10	0.4158	0.3897	0.9371
nanobeir / NanoClimateFEVER_cosine_map@100	0.2527	0.2201	0.8711
nanobeir / NanoDBPedia_cosine_accuracy@1	0.8200	0.7400	0.9024
nanobeir / NanoDBPedia_cosine_accuracy@3	0.9400	0.8800	0.9362
nanobeir / NanoDBPedia_cosine_accuracy@5	0.9400	0.9000	0.9574
nanobeir / NanoDBPedia_cosine_accuracy@10	0.9800	0.9800	1.0000
nanobeir / NanoDBPedia_cosine_precision@1	0.8200	0.7400	0.9024
nanobeir / NanoDBPedia_cosine_precision@3	0.6733	0.6200	0.9208
nanobeir / NanoDBPedia_cosine_precision@5	0.6160	0.5800	0.9416
nanobeir / NanoDBPedia_cosine_precision@10	0.5260	0.5080	0.9658
nanobeir / NanoDBPedia_cosine_recall@1	0.1157	0.0846	0.7316
nanobeir / NanoDBPedia_cosine_recall@3	0.2010	0.1740	0.8656
nanobeir / NanoDBPedia_cosine_recall@5	0.2554	0.2349	0.9200
nanobeir / NanoDBPedia_cosine_recall@10	0.3663	0.3450	0.9420
nanobeir / NanoDBPedia_cosine_ndcg@10	0.6663	0.6271	0.9412
nanobeir / NanoDBPedia_cosine_mrr@10	0.8767	0.8228	0.9386
nanobeir / NanoDBPedia_cosine_map@100	0.5239	0.4803	0.9167
nanobeir / NanoFEVER_cosine_accuracy@1	0.9200	0.8200	0.8913
nanobeir / NanoFEVER_cosine_accuracy@3	0.9800	0.9200	0.9388
nanobeir / NanoFEVER_cosine_accuracy@5	1.0000	0.9600	0.9600
nanobeir / NanoFEVER_cosine_accuracy@10	1.0000	0.9800	0.9800
nanobeir / NanoFEVER_cosine_precision@1	0.9200	0.8200	0.8913
nanobeir / NanoFEVER_cosine_precision@3	0.3467	0.3200	0.9231
nanobeir / NanoFEVER_cosine_precision@5	0.2120	0.2000	0.9434
nanobeir / NanoFEVER_cosine_precision@10	0.1060	0.1020	0.9623
nanobeir / NanoFEVER_cosine_recall@1	0.8567	0.7667	0.8949
nanobeir / NanoFEVER_cosine_recall@3	0.9433	0.8733	0.9258
nanobeir / NanoFEVER_cosine_recall@5	0.9633	0.9133	0.9481
nanobeir / NanoFEVER_cosine_recall@10	0.9633	0.9333	0.9689
nanobeir / NanoFEVER_cosine_ndcg@10	0.9359	0.8696	0.9291
nanobeir / NanoFEVER_cosine_mrr@10	0.9517	0.8792	0.9238
nanobeir / NanoFEVER_cosine_map@100	0.9184	0.8383	0.9127
nanobeir / NanoFiQA2018_cosine_accuracy@1	0.4600	0.4600	1.0000
nanobeir / NanoFiQA2018_cosine_accuracy@3	0.7200	0.6800	0.9444
nanobeir / NanoFiQA2018_cosine_accuracy@5	0.7600	0.7000	0.9211
nanobeir / NanoFiQA2018_cosine_accuracy@10	0.8200	0.7800	0.9512
nanobeir / NanoFiQA2018_cosine_precision@1	0.4600	0.4600	1.0000
nanobeir / NanoFiQA2018_cosine_precision@3	0.3467	0.2933	0.8462
nanobeir / NanoFiQA2018_cosine_precision@5	0.2760	0.2160	0.7826
nanobeir / NanoFiQA2018_cosine_precision@10	0.1560	0.1240	0.7949
nanobeir / NanoFiQA2018_cosine_recall@1	0.2527	0.2486	0.9835
nanobeir / NanoFiQA2018_cosine_recall@3	0.5127	0.4522	0.8821
nanobeir / NanoFiQA2018_cosine_recall@5	0.6196	0.5113	0.8253
nanobeir / NanoFiQA2018_cosine_recall@10	0.6842	0.5614	0.8205
nanobeir / NanoFiQA2018_cosine_ndcg@10	0.5704	0.4929	0.8641
nanobeir / NanoFiQA2018_cosine_mrr@10	0.5985	0.5815	0.9715
nanobeir / NanoFiQA2018_cosine_map@100	0.5011	0.4338	0.8658
nanobeir / NanoHotpotQA_cosine_accuracy@1	0.8400	0.8400	1.0000
nanobeir / NanoHotpotQA_cosine_accuracy@3	0.9400	0.9600	1.0213
nanobeir / NanoHotpotQA_cosine_accuracy@5	0.9800	0.9600	0.9796
nanobeir / NanoHotpotQA_cosine_accuracy@10	0.9800	0.9800	1.0000
nanobeir / NanoHotpotQA_cosine_precision@1	0.8400	0.8400	1.0000
nanobeir / NanoHotpotQA_cosine_precision@3	0.5200	0.5067	0.9744
nanobeir / NanoHotpotQA_cosine_precision@5	0.3360	0.3120	0.9286
nanobeir / NanoHotpotQA_cosine_precision@10	0.1780	0.1700	0.9551
nanobeir / NanoHotpotQA_cosine_recall@1	0.4200	0.4200	1.0000
nanobeir / NanoHotpotQA_cosine_recall@3	0.7800	0.7600	0.9744
nanobeir / NanoHotpotQA_cosine_recall@5	0.8400	0.7800	0.9286
nanobeir / NanoHotpotQA_cosine_recall@10	0.8900	0.8500	0.9551
nanobeir / NanoHotpotQA_cosine_ndcg@10	0.8322	0.8019	0.9636
nanobeir / NanoHotpotQA_cosine_mrr@10	0.8923	0.8958	1.0039
nanobeir / NanoHotpotQA_cosine_map@100	0.7766	0.7398	0.9526
nanobeir / NanoMSMARCO_cosine_accuracy@1	0.4400	0.4000	0.9091
nanobeir / NanoMSMARCO_cosine_accuracy@3	0.5600	0.5800	1.0357
nanobeir / NanoMSMARCO_cosine_accuracy@5	0.7000	0.7200	1.0286
nanobeir / NanoMSMARCO_cosine_accuracy@10	0.9000	0.9000	1.0000
nanobeir / NanoMSMARCO_cosine_precision@1	0.4400	0.4000	0.9091
nanobeir / NanoMSMARCO_cosine_precision@3	0.1867	0.1933	1.0357
nanobeir / NanoMSMARCO_cosine_precision@5	0.1400	0.1440	1.0286
nanobeir / NanoMSMARCO_cosine_precision@10	0.0900	0.0900	1.0000
nanobeir / NanoMSMARCO_cosine_recall@1	0.4400	0.4000	0.9091
nanobeir / NanoMSMARCO_cosine_recall@3	0.5600	0.5800	1.0357
nanobeir / NanoMSMARCO_cosine_recall@5	0.7000	0.7200	1.0286
nanobeir / NanoMSMARCO_cosine_recall@10	0.9000	0.9000	1.0000
nanobeir / NanoMSMARCO_cosine_ndcg@10	0.6329	0.6232	0.9847
nanobeir / NanoMSMARCO_cosine_mrr@10	0.5523	0.5382	0.9744
nanobeir / NanoMSMARCO_cosine_map@100	0.5566	0.5450	0.9792
nanobeir / NanoNFCorpus_cosine_accuracy@1	0.4800	0.4200	0.8750
nanobeir / NanoNFCorpus_cosine_accuracy@3	0.6200	0.5400	0.8710
nanobeir / NanoNFCorpus_cosine_accuracy@5	0.7000	0.6600	0.9429
nanobeir / NanoNFCorpus_cosine_accuracy@10	0.7600	0.7600	1.0000
nanobeir / NanoNFCorpus_cosine_precision@1	0.4800	0.4200	0.8750
nanobeir / NanoNFCorpus_cosine_precision@3	0.4267	0.3667	0.8594
nanobeir / NanoNFCorpus_cosine_precision@5	0.3840	0.3440	0.8958
nanobeir / NanoNFCorpus_cosine_precision@10	0.3360	0.2940	0.8750
nanobeir / NanoNFCorpus_cosine_recall@1	0.0273	0.0160	0.5856
nanobeir / NanoNFCorpus_cosine_recall@3	0.0660	0.0431	0.6532
nanobeir / NanoNFCorpus_cosine_recall@5	0.1303	0.0716	0.5495
nanobeir / NanoNFCorpus_cosine_recall@10	0.1725	0.1396	0.8089
nanobeir / NanoNFCorpus_cosine_ndcg@10	0.3927	0.3353	0.8538
nanobeir / NanoNFCorpus_cosine_mrr@10	0.5627	0.5154	0.9159
nanobeir / NanoNFCorpus_cosine_map@100	0.1746	0.1334	0.7641
nanobeir / NanoNQ_cosine_accuracy@1	0.6600	0.5200	0.7879
nanobeir / NanoNQ_cosine_accuracy@3	0.8200	0.6600	0.8049
nanobeir / NanoNQ_cosine_accuracy@5	0.8600	0.7800	0.9070
nanobeir / NanoNQ_cosine_accuracy@10	0.9200	0.8400	0.9130
nanobeir / NanoNQ_cosine_precision@1	0.6600	0.5200	0.7879
nanobeir / NanoNQ_cosine_precision@3	0.2800	0.2200	0.7857
nanobeir / NanoNQ_cosine_precision@5	0.1840	0.1560	0.8478
nanobeir / NanoNQ_cosine_precision@10	0.1000	0.0900	0.9000
nanobeir / NanoNQ_cosine_recall@1	0.6400	0.4900	0.7656
nanobeir / NanoNQ_cosine_recall@3	0.7700	0.6200	0.8052
nanobeir / NanoNQ_cosine_recall@5	0.8200	0.7200	0.8780
nanobeir / NanoNQ_cosine_recall@10	0.8900	0.8000	0.8989
nanobeir / NanoNQ_cosine_ndcg@10	0.7744	0.6479	0.8367
nanobeir / NanoNQ_cosine_mrr@10	0.7487	0.6189	0.8267
nanobeir / NanoNQ_cosine_map@100	0.7320	0.5969	0.8155
nanobeir / NanoQuoraRetrieval_cosine_accuracy@1	0.8600	0.9000	1.0465
nanobeir / NanoQuoraRetrieval_cosine_accuracy@3	0.9400	0.9400	1.0000
nanobeir / NanoQuoraRetrieval_cosine_accuracy@5	0.9600	0.9600	1.0000
nanobeir / NanoQuoraRetrieval_cosine_accuracy@10	0.9800	0.9800	1.0000
nanobeir / NanoQuoraRetrieval_cosine_precision@1	0.8600	0.9000	1.0465
nanobeir / NanoQuoraRetrieval_cosine_precision@3	0.3933	0.3733	0.9492
nanobeir / NanoQuoraRetrieval_cosine_precision@5	0.2440	0.2440	1.0000
nanobeir / NanoQuoraRetrieval_cosine_precision@10	0.1380	0.1360	0.9855
nanobeir / NanoQuoraRetrieval_cosine_recall@1	0.7573	0.7873	1.0396
nanobeir / NanoQuoraRetrieval_cosine_recall@3	0.9013	0.8907	0.9882
nanobeir / NanoQuoraRetrieval_cosine_recall@5	0.9247	0.9220	0.9971
nanobeir / NanoQuoraRetrieval_cosine_recall@10	0.9800	0.9767	0.9966
nanobeir / NanoQuoraRetrieval_cosine_ndcg@10	0.9172	0.9228	1.0061
nanobeir / NanoQuoraRetrieval_cosine_mrr@10	0.9003	0.9237	1.0259
nanobeir / NanoQuoraRetrieval_cosine_map@100	0.8933	0.8962	1.0033
nanobeir / NanoSCIDOCS_cosine_accuracy@1	0.5000	0.5000	1.0000
nanobeir / NanoSCIDOCS_cosine_accuracy@3	0.6600	0.7200	1.0909
nanobeir / NanoSCIDOCS_cosine_accuracy@5	0.7800	0.7800	1.0000
nanobeir / NanoSCIDOCS_cosine_accuracy@10	0.8400	0.8400	1.0000
nanobeir / NanoSCIDOCS_cosine_precision@1	0.5000	0.5000	1.0000
nanobeir / NanoSCIDOCS_cosine_precision@3	0.3667	0.3600	0.9818
nanobeir / NanoSCIDOCS_cosine_precision@5	0.3200	0.3040	0.9500
nanobeir / NanoSCIDOCS_cosine_precision@10	0.1960	0.1980	1.0102
nanobeir / NanoSCIDOCS_cosine_recall@1	0.1047	0.1057	1.0096
nanobeir / NanoSCIDOCS_cosine_recall@3	0.2267	0.2217	0.9779
nanobeir / NanoSCIDOCS_cosine_recall@5	0.3287	0.3117	0.9483
nanobeir / NanoSCIDOCS_cosine_recall@10	0.4017	0.4047	1.0075
nanobeir / NanoSCIDOCS_cosine_ndcg@10	0.4028	0.4056	1.0069
nanobeir / NanoSCIDOCS_cosine_mrr@10	0.6088	0.6200	1.0184
nanobeir / NanoSCIDOCS_cosine_map@100	0.3195	0.3145	0.9844
nanobeir / NanoArguAna_cosine_accuracy@1	0.2800	0.2400	0.8571
nanobeir / NanoArguAna_cosine_accuracy@3	0.7000	0.6600	0.9429
nanobeir / NanoArguAna_cosine_accuracy@5	0.8000	0.8000	1.0000
nanobeir / NanoArguAna_cosine_accuracy@10	0.9400	0.9000	0.9574
nanobeir / NanoArguAna_cosine_precision@1	0.2800	0.2400	0.8571
nanobeir / NanoArguAna_cosine_precision@3	0.2333	0.2200	0.9429
nanobeir / NanoArguAna_cosine_precision@5	0.1600	0.1600	1.0000
nanobeir / NanoArguAna_cosine_precision@10	0.0940	0.0900	0.9574
nanobeir / NanoArguAna_cosine_recall@1	0.2800	0.2400	0.8571
nanobeir / NanoArguAna_cosine_recall@3	0.7000	0.6600	0.9429
nanobeir / NanoArguAna_cosine_recall@5	0.8000	0.8000	1.0000
nanobeir / NanoArguAna_cosine_recall@10	0.9400	0.9000	0.9574
nanobeir / NanoArguAna_cosine_ndcg@10	0.6187	0.5824	0.9414
nanobeir / NanoArguAna_cosine_mrr@10	0.5146	0.4793	0.9314
nanobeir / NanoArguAna_cosine_map@100	0.5185	0.4851	0.9356
nanobeir / NanoSciFact_cosine_accuracy@1	0.7600	0.6400	0.8421
nanobeir / NanoSciFact_cosine_accuracy@3	0.9200	0.8400	0.9130
nanobeir / NanoSciFact_cosine_accuracy@5	0.9400	0.8800	0.9362
nanobeir / NanoSciFact_cosine_accuracy@10	0.9400	0.9400	1.0000
nanobeir / NanoSciFact_cosine_precision@1	0.7600	0.6400	0.8421
nanobeir / NanoSciFact_cosine_precision@3	0.3333	0.3000	0.9000
nanobeir / NanoSciFact_cosine_precision@5	0.2080	0.1960	0.9423
nanobeir / NanoSciFact_cosine_precision@10	0.1060	0.1040	0.9811
nanobeir / NanoSciFact_cosine_recall@1	0.7250	0.6050	0.8345
nanobeir / NanoSciFact_cosine_recall@3	0.9050	0.8300	0.9171
nanobeir / NanoSciFact_cosine_recall@5	0.9300	0.8800	0.9462
nanobeir / NanoSciFact_cosine_recall@10	0.9400	0.9300	0.9894
nanobeir / NanoSciFact_cosine_ndcg@10	0.8613	0.7943	0.9222
nanobeir / NanoSciFact_cosine_mrr@10	0.8340	0.7515	0.9011
nanobeir / NanoSciFact_cosine_map@100	0.8351	0.7497	0.8977
nanobeir / NanoTouche2020_cosine_accuracy@1	0.7143	0.6939	0.9714
nanobeir / NanoTouche2020_cosine_accuracy@3	0.9388	0.8776	0.9348
nanobeir / NanoTouche2020_cosine_accuracy@5	0.9592	0.9184	0.9574
nanobeir / NanoTouche2020_cosine_accuracy@10	1.0000	1.0000	1.0000
nanobeir / NanoTouche2020_cosine_precision@1	0.7143	0.6939	0.9714
nanobeir / NanoTouche2020_cosine_precision@3	0.7143	0.6122	0.8571
nanobeir / NanoTouche2020_cosine_precision@5	0.6367	0.6000	0.9423
nanobeir / NanoTouche2020_cosine_precision@10	0.5204	0.4816	0.9255
nanobeir / NanoTouche2020_cosine_recall@1	0.0523	0.0506	0.9672
nanobeir / NanoTouche2020_cosine_recall@3	0.1518	0.1351	0.8902
nanobeir / NanoTouche2020_cosine_recall@5	0.2182	0.2121	0.9719
nanobeir / NanoTouche2020_cosine_recall@10	0.3432	0.3266	0.9516
nanobeir / NanoTouche2020_cosine_ndcg@10	0.5914	0.5483	0.9271
nanobeir / NanoTouche2020_cosine_mrr@10	0.8193	0.7951	0.9704
nanobeir / NanoTouche2020_cosine_map@100	0.4510	0.4399	0.9753
nanobeir / NanoBEIR_mean_cosine_accuracy@1	0.6180	0.5734	0.9278
nanobeir / NanoBEIR_mean_cosine_accuracy@3	0.7830	0.7444	0.9508
nanobeir / NanoBEIR_mean_cosine_accuracy@5	0.8461	0.8137	0.9617
nanobeir / NanoBEIR_mean_cosine_accuracy@10	0.9123	0.8923	0.9781
nanobeir / NanoBEIR_mean_cosine_precision@1	0.6180	0.5734	0.9278
nanobeir / NanoBEIR_mean_cosine_precision@3	0.3842	0.3502	0.9115
nanobeir / NanoBEIR_mean_cosine_precision@5	0.2970	0.2766	0.9314
nanobeir / NanoBEIR_mean_cosine_precision@10	0.2043	0.1912	0.9357
nanobeir / NanoBEIR_mean_cosine_recall@1	0.3700	0.3321	0.8977
nanobeir / NanoBEIR_mean_cosine_recall@3	0.5350	0.4959	0.9270
nanobeir / NanoBEIR_mean_cosine_recall@5	0.6037	0.5660	0.9376
nanobeir / NanoBEIR_mean_cosine_recall@10	0.6856	0.6491	0.9468
nanobeir / NanoBEIR_mean_cosine_ndcg@10	0.6562	0.6109	0.9310
nanobeir / NanoBEIR_mean_cosine_mrr@10	0.7135	0.6778	0.9499
nanobeir / NanoBEIR_mean_cosine_map@100	0.5733	0.5287	0.9221

Citation

If you use this model or the pruning approach, please cite:

bibtex

@misc{subedi2025tokenpruning,
  author = {Sanjaya Subedi},
  title  = {Token Embedding Pruning for Sentence Transformers},
  year   = {2026},
  note   = {Available at: https://sanjayasubedi.com.np/deeplearning/shrinking-embedding-models-by-pruning-vocabulary/}
}