Token-embedding pruning clusters semantically similar tokens in the embedding space (using DBSCAN) and merges each cluster into a single shared embedding, shrinking the vocabulary and reducing memory without retraining the transformer layers.

How to use

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jangedoo/embeddinggemma-300m-ne-pruned", 
                            trust_remote_code=True)
embeddings = model.encode(["Hello world", "How are you?"])

Note: trust_remote_code=True is required because the model ships a small custom tokenizer class (pruned_tokenizer.py) that applies the id remapping after tokenization. No additional package installation is needed.

Pruning statistics

	Base	Pruned	Reduction
Vocab size	262,144	28,195	89.24%
Total parameters	307,581,696	127,908,864	58.41%
Embedding parameters	201,326,592	21,653,760	89.24%
Embedding size (MB)	768.0	82.6	685.4 MB saved

Evaluation

Dataset / Metric	Base	Pruned	Relative (base = 1.0)
stsb_ne / stsb_ne_pearson_cosine	0.6941	0.6893	0.9931
stsb_ne / stsb_ne_spearman_cosine	0.6831	0.6786	0.9933
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@1	0.2600	0.0800	0.3077
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@3	0.3800	0.2600	0.6842
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@5	0.5800	0.3800	0.6552
nanobeir_ne / NanoClimateFEVER_cosine_accuracy@10	0.7000	0.5200	0.7429
nanobeir_ne / NanoClimateFEVER_cosine_precision@1	0.2600	0.0800	0.3077
nanobeir_ne / NanoClimateFEVER_cosine_precision@3	0.1267	0.0867	0.6842
nanobeir_ne / NanoClimateFEVER_cosine_precision@5	0.1160	0.0760	0.6552
nanobeir_ne / NanoClimateFEVER_cosine_precision@10	0.0940	0.0620	0.6596
nanobeir_ne / NanoClimateFEVER_cosine_recall@1	0.1383	0.0400	0.2892
nanobeir_ne / NanoClimateFEVER_cosine_recall@3	0.1933	0.1233	0.6379
nanobeir_ne / NanoClimateFEVER_cosine_recall@5	0.2540	0.1600	0.6299
nanobeir_ne / NanoClimateFEVER_cosine_recall@10	0.3720	0.2440	0.6559
nanobeir_ne / NanoClimateFEVER_cosine_ndcg@10	0.2929	0.1651	0.5638
nanobeir_ne / NanoClimateFEVER_cosine_mrr@10	0.3779	0.2006	0.5307
nanobeir_ne / NanoClimateFEVER_cosine_map@100	0.2189	0.1193	0.5450
nanobeir_ne / NanoDBPedia_cosine_accuracy@1	0.6800	0.6200	0.9118
nanobeir_ne / NanoDBPedia_cosine_accuracy@3	0.8800	0.8000	0.9091
nanobeir_ne / NanoDBPedia_cosine_accuracy@5	0.9200	0.8600	0.9348
nanobeir_ne / NanoDBPedia_cosine_accuracy@10	0.9600	0.9400	0.9792
nanobeir_ne / NanoDBPedia_cosine_precision@1	0.6800	0.6200	0.9118
nanobeir_ne / NanoDBPedia_cosine_precision@3	0.5733	0.4400	0.7674
nanobeir_ne / NanoDBPedia_cosine_precision@5	0.5040	0.4040	0.8016
nanobeir_ne / NanoDBPedia_cosine_precision@10	0.4440	0.3640	0.8198
nanobeir_ne / NanoDBPedia_cosine_recall@1	0.0982	0.0785	0.7994
nanobeir_ne / NanoDBPedia_cosine_recall@3	0.1690	0.1227	0.7258
nanobeir_ne / NanoDBPedia_cosine_recall@5	0.2122	0.1680	0.7917
nanobeir_ne / NanoDBPedia_cosine_recall@10	0.3264	0.2418	0.7409
nanobeir_ne / NanoDBPedia_cosine_ndcg@10	0.5691	0.4533	0.7966
nanobeir_ne / NanoDBPedia_cosine_mrr@10	0.7952	0.7277	0.9152
nanobeir_ne / NanoDBPedia_cosine_map@100	0.4155	0.3202	0.7707
nanobeir_ne / NanoFEVER_cosine_accuracy@1	0.7400	0.3600	0.4865
nanobeir_ne / NanoFEVER_cosine_accuracy@3	0.9200	0.7400	0.8043
nanobeir_ne / NanoFEVER_cosine_accuracy@5	0.9400	0.7800	0.8298
nanobeir_ne / NanoFEVER_cosine_accuracy@10	0.9600	0.8400	0.8750
nanobeir_ne / NanoFEVER_cosine_precision@1	0.7400	0.3600	0.4865
nanobeir_ne / NanoFEVER_cosine_precision@3	0.3133	0.2533	0.8085
nanobeir_ne / NanoFEVER_cosine_precision@5	0.1920	0.1600	0.8333
nanobeir_ne / NanoFEVER_cosine_precision@10	0.1000	0.0860	0.8600
nanobeir_ne / NanoFEVER_cosine_recall@1	0.7167	0.3600	0.5023
nanobeir_ne / NanoFEVER_cosine_recall@3	0.8767	0.7167	0.8175
nanobeir_ne / NanoFEVER_cosine_recall@5	0.8967	0.7467	0.8327
nanobeir_ne / NanoFEVER_cosine_recall@10	0.9267	0.7967	0.8597
nanobeir_ne / NanoFEVER_cosine_ndcg@10	0.8399	0.6019	0.7167
nanobeir_ne / NanoFEVER_cosine_mrr@10	0.8308	0.5425	0.6530
nanobeir_ne / NanoFEVER_cosine_map@100	0.8043	0.5388	0.6699
nanobeir_ne / NanoFiQA2018_cosine_accuracy@1	0.3400	0.2600	0.7647
nanobeir_ne / NanoFiQA2018_cosine_accuracy@3	0.5800	0.4800	0.8276
nanobeir_ne / NanoFiQA2018_cosine_accuracy@5	0.6200	0.5600	0.9032
nanobeir_ne / NanoFiQA2018_cosine_accuracy@10	0.6600	0.6200	0.9394
nanobeir_ne / NanoFiQA2018_cosine_precision@1	0.3400	0.2600	0.7647
nanobeir_ne / NanoFiQA2018_cosine_precision@3	0.2533	0.2267	0.8947
nanobeir_ne / NanoFiQA2018_cosine_precision@5	0.1800	0.1680	0.9333
nanobeir_ne / NanoFiQA2018_cosine_precision@10	0.1020	0.0980	0.9608
nanobeir_ne / NanoFiQA2018_cosine_recall@1	0.2075	0.1475	0.7109
nanobeir_ne / NanoFiQA2018_cosine_recall@3	0.4017	0.3551	0.8838
nanobeir_ne / NanoFiQA2018_cosine_recall@5	0.4506	0.4092	0.9081
nanobeir_ne / NanoFiQA2018_cosine_recall@10	0.5021	0.4625	0.9212
nanobeir_ne / NanoFiQA2018_cosine_ndcg@10	0.4213	0.3641	0.8642
nanobeir_ne / NanoFiQA2018_cosine_mrr@10	0.4617	0.3815	0.8262
nanobeir_ne / NanoFiQA2018_cosine_map@100	0.3688	0.3096	0.8395
nanobeir_ne / NanoHotpotQA_cosine_accuracy@1	0.8000	0.4600	0.5750
nanobeir_ne / NanoHotpotQA_cosine_accuracy@3	0.9400	0.6400	0.6809
nanobeir_ne / NanoHotpotQA_cosine_accuracy@5	0.9600	0.6800	0.7083
nanobeir_ne / NanoHotpotQA_cosine_accuracy@10	0.9800	0.7600	0.7755
nanobeir_ne / NanoHotpotQA_cosine_precision@1	0.8000	0.4600	0.5750
nanobeir_ne / NanoHotpotQA_cosine_precision@3	0.4333	0.2867	0.6615
nanobeir_ne / NanoHotpotQA_cosine_precision@5	0.2840	0.1800	0.6338
nanobeir_ne / NanoHotpotQA_cosine_precision@10	0.1640	0.1080	0.6585
nanobeir_ne / NanoHotpotQA_cosine_recall@1	0.4000	0.2300	0.5750
nanobeir_ne / NanoHotpotQA_cosine_recall@3	0.6500	0.4300	0.6615
nanobeir_ne / NanoHotpotQA_cosine_recall@5	0.7100	0.4500	0.6338
nanobeir_ne / NanoHotpotQA_cosine_recall@10	0.8200	0.5400	0.6585
nanobeir_ne / NanoHotpotQA_cosine_ndcg@10	0.7490	0.4754	0.6347
nanobeir_ne / NanoHotpotQA_cosine_mrr@10	0.8712	0.5607	0.6437
nanobeir_ne / NanoHotpotQA_cosine_map@100	0.6656	0.4212	0.6327
nanobeir_ne / NanoMSMARCO_cosine_accuracy@1	0.2800	0.2400	0.8571
nanobeir_ne / NanoMSMARCO_cosine_accuracy@3	0.5000	0.4800	0.9600
nanobeir_ne / NanoMSMARCO_cosine_accuracy@5	0.6400	0.5800	0.9062
nanobeir_ne / NanoMSMARCO_cosine_accuracy@10	0.7800	0.7000	0.8974
nanobeir_ne / NanoMSMARCO_cosine_precision@1	0.2800	0.2400	0.8571
nanobeir_ne / NanoMSMARCO_cosine_precision@3	0.1667	0.1600	0.9600
nanobeir_ne / NanoMSMARCO_cosine_precision@5	0.1280	0.1160	0.9063
nanobeir_ne / NanoMSMARCO_cosine_precision@10	0.0780	0.0700	0.8974
nanobeir_ne / NanoMSMARCO_cosine_recall@1	0.2800	0.2400	0.8571
nanobeir_ne / NanoMSMARCO_cosine_recall@3	0.5000	0.4800	0.9600
nanobeir_ne / NanoMSMARCO_cosine_recall@5	0.6400	0.5800	0.9062
nanobeir_ne / NanoMSMARCO_cosine_recall@10	0.7800	0.7000	0.8974
nanobeir_ne / NanoMSMARCO_cosine_ndcg@10	0.5131	0.4593	0.8950
nanobeir_ne / NanoMSMARCO_cosine_mrr@10	0.4300	0.3832	0.8913
nanobeir_ne / NanoMSMARCO_cosine_map@100	0.4375	0.3900	0.8914
nanobeir_ne / NanoNFCorpus_cosine_accuracy@1	0.3200	0.2200	0.6875
nanobeir_ne / NanoNFCorpus_cosine_accuracy@3	0.4200	0.3800	0.9048
nanobeir_ne / NanoNFCorpus_cosine_accuracy@5	0.4600	0.4200	0.9130
nanobeir_ne / NanoNFCorpus_cosine_accuracy@10	0.5800	0.5400	0.9310
nanobeir_ne / NanoNFCorpus_cosine_precision@1	0.3200	0.2200	0.6875
nanobeir_ne / NanoNFCorpus_cosine_precision@3	0.2867	0.2267	0.7907
nanobeir_ne / NanoNFCorpus_cosine_precision@5	0.2360	0.2120	0.8983
nanobeir_ne / NanoNFCorpus_cosine_precision@10	0.1920	0.1840	0.9583
nanobeir_ne / NanoNFCorpus_cosine_recall@1	0.0326	0.0279	0.8546
nanobeir_ne / NanoNFCorpus_cosine_recall@3	0.0517	0.0435	0.8416
nanobeir_ne / NanoNFCorpus_cosine_recall@5	0.0642	0.0555	0.8640
nanobeir_ne / NanoNFCorpus_cosine_recall@10	0.0997	0.0735	0.7367
nanobeir_ne / NanoNFCorpus_cosine_ndcg@10	0.2358	0.2095	0.8886
nanobeir_ne / NanoNFCorpus_cosine_mrr@10	0.3792	0.3106	0.8191
nanobeir_ne / NanoNFCorpus_cosine_map@100	0.0977	0.0824	0.8441
nanobeir_ne / NanoNQ_cosine_accuracy@1	0.3600	0.2200	0.6111
nanobeir_ne / NanoNQ_cosine_accuracy@3	0.5000	0.3000	0.6000
nanobeir_ne / NanoNQ_cosine_accuracy@5	0.5600	0.4600	0.8214
nanobeir_ne / NanoNQ_cosine_accuracy@10	0.7200	0.6400	0.8889
nanobeir_ne / NanoNQ_cosine_precision@1	0.3600	0.2200	0.6111
nanobeir_ne / NanoNQ_cosine_precision@3	0.1667	0.1000	0.6000
nanobeir_ne / NanoNQ_cosine_precision@5	0.1120	0.0920	0.8214
nanobeir_ne / NanoNQ_cosine_precision@10	0.0760	0.0640	0.8421
nanobeir_ne / NanoNQ_cosine_recall@1	0.3500	0.2100	0.6000
nanobeir_ne / NanoNQ_cosine_recall@3	0.4800	0.2900	0.6042
nanobeir_ne / NanoNQ_cosine_recall@5	0.5400	0.4300	0.7963
nanobeir_ne / NanoNQ_cosine_recall@10	0.6800	0.6000	0.8824
nanobeir_ne / NanoNQ_cosine_ndcg@10	0.5024	0.3761	0.7487
nanobeir_ne / NanoNQ_cosine_mrr@10	0.4548	0.3174	0.6980
nanobeir_ne / NanoNQ_cosine_map@100	0.4489	0.3158	0.7036
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@1	0.8600	0.8200	0.9535
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@3	0.9000	0.9000	1.0000
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@5	0.9200	0.9600	1.0435
nanobeir_ne / NanoQuoraRetrieval_cosine_accuracy@10	1.0000	0.9800	0.9800
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@1	0.8600	0.8200	0.9535
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@3	0.3533	0.3600	1.0189
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@5	0.2320	0.2360	1.0172
nanobeir_ne / NanoQuoraRetrieval_cosine_precision@10	0.1280	0.1300	1.0156
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@1	0.7373	0.7373	1.0000
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@3	0.8440	0.8620	1.0213
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@5	0.8887	0.9160	1.0308
nanobeir_ne / NanoQuoraRetrieval_cosine_recall@10	0.9767	0.9667	0.9898
nanobeir_ne / NanoQuoraRetrieval_cosine_ndcg@10	0.8892	0.8883	0.9989
nanobeir_ne / NanoQuoraRetrieval_cosine_mrr@10	0.8907	0.8692	0.9758
nanobeir_ne / NanoQuoraRetrieval_cosine_map@100	0.8548	0.8576	1.0033
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@1	0.3400	0.2800	0.8235
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@3	0.4800	0.4200	0.8750
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@5	0.6000	0.5400	0.9000
nanobeir_ne / NanoSCIDOCS_cosine_accuracy@10	0.6600	0.6600	1.0000
nanobeir_ne / NanoSCIDOCS_cosine_precision@1	0.3400	0.2800	0.8235
nanobeir_ne / NanoSCIDOCS_cosine_precision@3	0.2333	0.2000	0.8571
nanobeir_ne / NanoSCIDOCS_cosine_precision@5	0.1880	0.1680	0.8936
nanobeir_ne / NanoSCIDOCS_cosine_precision@10	0.1220	0.1260	1.0328
nanobeir_ne / NanoSCIDOCS_cosine_recall@1	0.0700	0.0570	0.8143
nanobeir_ne / NanoSCIDOCS_cosine_recall@3	0.1430	0.1210	0.8462
nanobeir_ne / NanoSCIDOCS_cosine_recall@5	0.1920	0.1710	0.8906
nanobeir_ne / NanoSCIDOCS_cosine_recall@10	0.2490	0.2560	1.0281
nanobeir_ne / NanoSCIDOCS_cosine_ndcg@10	0.2539	0.2411	0.9495
nanobeir_ne / NanoSCIDOCS_cosine_mrr@10	0.4419	0.3897	0.8821
nanobeir_ne / NanoSCIDOCS_cosine_map@100	0.1912	0.1712	0.8955
nanobeir_ne / NanoArguAna_cosine_accuracy@1	0.2000	0.1800	0.9000
nanobeir_ne / NanoArguAna_cosine_accuracy@3	0.5400	0.5400	1.0000
nanobeir_ne / NanoArguAna_cosine_accuracy@5	0.6200	0.6400	1.0323
nanobeir_ne / NanoArguAna_cosine_accuracy@10	0.7800	0.7600	0.9744
nanobeir_ne / NanoArguAna_cosine_precision@1	0.2000	0.1800	0.9000
nanobeir_ne / NanoArguAna_cosine_precision@3	0.1800	0.1800	1.0000
nanobeir_ne / NanoArguAna_cosine_precision@5	0.1240	0.1280	1.0323
nanobeir_ne / NanoArguAna_cosine_precision@10	0.0780	0.0760	0.9744
nanobeir_ne / NanoArguAna_cosine_recall@1	0.2000	0.1800	0.9000
nanobeir_ne / NanoArguAna_cosine_recall@3	0.5400	0.5400	1.0000
nanobeir_ne / NanoArguAna_cosine_recall@5	0.6200	0.6400	1.0323
nanobeir_ne / NanoArguAna_cosine_recall@10	0.7800	0.7600	0.9744
nanobeir_ne / NanoArguAna_cosine_ndcg@10	0.4864	0.4683	0.9628
nanobeir_ne / NanoArguAna_cosine_mrr@10	0.3927	0.3751	0.9550
nanobeir_ne / NanoArguAna_cosine_map@100	0.4014	0.3852	0.9597
nanobeir_ne / NanoSciFact_cosine_accuracy@1	0.4000	0.4000	1.0000
nanobeir_ne / NanoSciFact_cosine_accuracy@3	0.6000	0.5200	0.8667
nanobeir_ne / NanoSciFact_cosine_accuracy@5	0.6800	0.5400	0.7941
nanobeir_ne / NanoSciFact_cosine_accuracy@10	0.8000	0.6600	0.8250
nanobeir_ne / NanoSciFact_cosine_precision@1	0.4000	0.4000	1.0000
nanobeir_ne / NanoSciFact_cosine_precision@3	0.2133	0.1933	0.9062
nanobeir_ne / NanoSciFact_cosine_precision@5	0.1480	0.1240	0.8378
nanobeir_ne / NanoSciFact_cosine_precision@10	0.0900	0.0760	0.8444
nanobeir_ne / NanoSciFact_cosine_recall@1	0.3800	0.3800	1.0000
nanobeir_ne / NanoSciFact_cosine_recall@3	0.5750	0.5100	0.8870
nanobeir_ne / NanoSciFact_cosine_recall@5	0.6600	0.5350	0.8106
nanobeir_ne / NanoSciFact_cosine_recall@10	0.7900	0.6600	0.8354
nanobeir_ne / NanoSciFact_cosine_ndcg@10	0.5811	0.5180	0.8915
nanobeir_ne / NanoSciFact_cosine_mrr@10	0.5180	0.4727	0.9127
nanobeir_ne / NanoSciFact_cosine_map@100	0.5170	0.4799	0.9283
nanobeir_ne / NanoTouche2020_cosine_accuracy@1	0.7347	0.4898	0.6667
nanobeir_ne / NanoTouche2020_cosine_accuracy@3	0.8571	0.7347	0.8571
nanobeir_ne / NanoTouche2020_cosine_accuracy@5	0.8980	0.8571	0.9545
nanobeir_ne / NanoTouche2020_cosine_accuracy@10	0.9388	0.9184	0.9783
nanobeir_ne / NanoTouche2020_cosine_precision@1	0.7347	0.4898	0.6667
nanobeir_ne / NanoTouche2020_cosine_precision@3	0.5306	0.4558	0.8590
nanobeir_ne / NanoTouche2020_cosine_precision@5	0.4980	0.4408	0.8852
nanobeir_ne / NanoTouche2020_cosine_precision@10	0.4122	0.3735	0.9059
nanobeir_ne / NanoTouche2020_cosine_recall@1	0.0476	0.0325	0.6819
nanobeir_ne / NanoTouche2020_cosine_recall@3	0.1063	0.0896	0.8424
nanobeir_ne / NanoTouche2020_cosine_recall@5	0.1666	0.1387	0.8323
nanobeir_ne / NanoTouche2020_cosine_recall@10	0.2660	0.2374	0.8924
nanobeir_ne / NanoTouche2020_cosine_ndcg@10	0.4785	0.4098	0.8565
nanobeir_ne / NanoTouche2020_cosine_mrr@10	0.8075	0.6399	0.7924
nanobeir_ne / NanoTouche2020_cosine_map@100	0.3664	0.3124	0.8526
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@1	0.4857	0.3561	0.7332
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@3	0.6536	0.5534	0.8467
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@5	0.7229	0.6352	0.8786
nanobeir_ne / NanoBEIR_mean_cosine_accuracy@10	0.8091	0.7337	0.9068
nanobeir_ne / NanoBEIR_mean_cosine_precision@1	0.4857	0.3561	0.7332
nanobeir_ne / NanoBEIR_mean_cosine_precision@3	0.2947	0.2438	0.8273
nanobeir_ne / NanoBEIR_mean_cosine_precision@5	0.2263	0.1927	0.8514
nanobeir_ne / NanoBEIR_mean_cosine_precision@10	0.1600	0.1398	0.8737
nanobeir_ne / NanoBEIR_mean_cosine_recall@1	0.2814	0.2093	0.7437
nanobeir_ne / NanoBEIR_mean_cosine_recall@3	0.4254	0.3603	0.8469
nanobeir_ne / NanoBEIR_mean_cosine_recall@5	0.4842	0.4154	0.8578
nanobeir_ne / NanoBEIR_mean_cosine_recall@10	0.5822	0.5030	0.8639
nanobeir_ne / NanoBEIR_mean_cosine_ndcg@10	0.5241	0.4331	0.8264
nanobeir_ne / NanoBEIR_mean_cosine_mrr@10	0.5886	0.4747	0.8065
nanobeir_ne / NanoBEIR_mean_cosine_map@100	0.4452	0.3618	0.8127

Citation

If you use this model or the pruning approach, please cite:

bibtex

@misc{subedi2025tokenpruning,
  author = {Sanjaya Subedi},
  title  = {Token Embedding Pruning for Sentence Transformers},
  year   = {2026},
  note   = {Available at: https://sanjayasubedi.com.np/deeplearning/shrinking-embedding-models-by-pruning-vocabulary/}
}