Total vocabulary size: 32768 tokens (reduced from 250002)
Tokenizer type: Unigram
Training samples per language: 200000 texts
Dataset: Lumberjackk/fineweb-2-trimming

Language Distribution

pms: 32768 tokens

This pruned model should perform similarly to the original model for 32768 with a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.

Usage

You can use this model with the Transformers library:

python

from transformers import AutoModel, AutoTokenizer

model_name = "Lumberjackk/multilingual-e5-large-pms-32768"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Model Statistics

Original model size: 559.9M parameters
Pruned model size: 337.4M parameters
Size reduction: 39.7%
Vocabulary reduction: 86.9%

alphaedge-ai/multilingual-e5-large-pms-32768

multilingual-e5-large-pms-32768

Language Distribution

Usage

Model Statistics

alphaedge-ai/
multilingual-e5-large-pms-32768