Total vocabulary size: 16384 tokens (reduced from 250002)
Tokenizer type: Unigram
Training samples per language: 200000 texts
Dataset: Lumberjackk/fineweb-2-trimming

Language Distribution

pms: 16384 tokens

This pruned model should perform similarly to the original model for 16384 with a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.

Usage

You can use this model with the Transformers library:

python

from transformers import AutoModel, AutoTokenizer

model_name = "Lumberjackk/multilingual-e5-large-pms-16384"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Model Statistics

Original model size: 559.9M parameters
Pruned model size: 320.7M parameters
Size reduction: 42.7%
Vocabulary reduction: 93.4%

alphaedge-ai/multilingual-e5-large-pms-16384

multilingual-e5-large-pms-16384

Language Distribution

Usage

Model Statistics

alphaedge-ai/
multilingual-e5-large-pms-16384