Name: fihris-embeddding-id-ar
Availability: InStock

You can deploy this model and start earning money today!

Model Summary

This project provides a universal tokenizer pruning and multilingual continual pre-training pipeline built on top of sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. The pipeline performs:

Tokenizer Pruning — Removes unused tokens from a multilingual Unigram tokenizer to create a compact vocabulary optimized for specific language(s).
Embedding Pruning — Correctly resizes token embeddings using resize_token_embeddings() with ID remapping, avoiding the common pitfall of shape-mismatched state dict loading.
Multilingual Continual Pre-training — Further trains the pruned model on Indonesian and Arabic corpora using Masked Language Modeling (MLM).

The result is a lightweight, language-specific sentence embedding model with significantly reduced vocabulary size (from ~250K to ~39K tokens) while preserving semantic quality across target languages.

Intended Uses & Limitations

Intended Uses

Semantic Search for Indonesian and Arabic text
Text Clustering and similarity tasks
Cross-lingual retrieval between Indonesian ↔ Arabic ↔ English
Resource-constrained deployments where smaller vocabularies are preferred

Limitations

The pruned vocabulary is optimized for the target language(s) selected during pruning; performance may degrade on languages outside the training distribution.
Embedding quality is verified via cosine similarity comparison between original and pruned models, but some semantic drift is expected.
The model is a sentence transformer (384-dim embeddings), not a generative model.

Training Details

Stage 1: Tokenizer Pruning

Base Tokenizer: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (Unigram, 250K vocab)
Pruning Method: Frequency-based token removal using FineWeb-2 corpus
Supported Presets:
- en_only — English only
- id_only — Indonesian only
- en_id — English + Indonesian
- id_ar — Indonesian + Arabic
- ar_only — Arabic only
- en_id_ar — English + Indonesian + Arabic
Metrics Tracked:
- BPT (Bytes Per Token) — higher is more efficient
- Perplexity (PPL) — lower is better
- Embedding Cosine Similarity — measures semantic preservation

Stage 2: Continual Pre-training (MLM)

Base Model: Ik45/fihris-embeddding-id-ar (derived from the pruned multilingual MiniLM)
Training Data:
- Arabic: ik4545/shamela-waqfeya-split (classical Arabic texts, ~3.57GB)
- Indonesian: Custom modern Indonesian text corpus (indo_modern.txt)
Training Objective: Masked Language Modeling (MLM)
- Mask probability: 15%
- 80% → [MASK], 10% → random token, 10% → unchanged
Hyperparameters:
- Epochs: 2
- Batch size: 16
- Learning rate: 1e-3
- Warmup ratio: 0.1
- Weight decay: 0.01
- Max sequence length: 512
- Mixed precision: FP16
Data Collator: Custom DataCollatorForMLMOnTheFly with on-the-fly tokenization

Stage 3: MLM Head Reconstruction (for Embedding → MLM)

When converting a sentence transformer encoder to an MLM model:

Encoder weights are copied from the pre-trained model
Decoder weight is tied to the word embedding matrix
Transform layer initialized as near-identity mapping
LayerNorm and decoder bias initialized to standard values

Evaluation

Quality Verification Metrics

Language	BPT Original	BPT Pruned	PPL Original	PPL Pruned	CosSim
Target	~3.5	~4.2	~12.3	~14.1	>0.95
Other	~3.5	~2.8	~12.3	~18.5	~0.85

Values are illustrative; actual results depend on the selected language preset.

Evaluation Methodology

BPT measured on held-out corpus samples
PPL computed via forward pass on masked sequences
Cosine Similarity between original and pruned model embeddings on identical sentences

How to Use

Quick Start — Sentence Embeddings

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("your-username/fihris-embedding-id-ar-pruned")
sentences = [
    "Ini adalah kalimat dalam bahasa Indonesia.",
    "هذه جملة باللغة العربية.",
    "This is a sentence in English."
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 384)

Using with HuggingFace Transformers

python

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("your-username/fihris-embedding-id-ar-pruned")
model = AutoModel.from_pretrained("your-username/fihris-embedding-id-ar-pruned")

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output.last_hidden_state
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

texts = ["السلام عليكم", "Selamat pagi", "Good morning"]
encoded = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")

with torch.no_grad():
    output = model(**encoded)

embeddings = mean_pooling(output, encoded["attention_mask"])

Running the Pruning Pipeline

python

# Configure and run the pruning notebook
UNI_MODEL_NAME = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
UNI_PRESET = "id_ar"  # Indonesian + Arabic
UNI_PRUNE_EMBEDDINGS = True
UNI_OUTPUT_DIR = "./pruned_output"

# The notebook will:
# 1. Load tokenizer and detect type (Unigram/BPE/WordPiece)
# 2. Compute token frequencies from FineWeb-2
# 3. Remove low-frequency tokens
# 4. Prune embeddings via resize_token_embeddings()
# 5. Verify quality with BPT, PPL, and CosSim
# 6. Save pruned tokenizer + model

Citation

If you use this model or pipeline, please cite:

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "http://arxiv.org/abs/1908.10084",
}

Acknowledgments

Base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
Arabic corpus: ik4545/shamela-waqfeya-split (Shamela Waqfeya)
Training corpora: HuggingFaceFW/fineweb and HuggingFaceFW/fineweb-2
Framework: HuggingFace Transformers, Sentence-Transformers

Ik45/fihris-embeddding-id-ar

Model Summary

Intended Uses & Limitations

Intended Uses

Limitations

Training Details

Stage 1: Tokenizer Pruning

Stage 2: Continual Pre-training (MLM)

Stage 3: MLM Head Reconstruction (for Embedding → MLM)

Evaluation

Quality Verification Metrics

Evaluation Methodology

How to Use

Quick Start — Sentence Embeddings

Ik45/
fihris-embeddding-id-ar