Views
No views yet
TigreGotico/rawi.TigreGotico/arabic_diacritized_text:| metric | value |
|---|---|
| DER (all positions) | 2.29% |
| DER* (marked positions) | 3.37% |
| WER | 8.33% |
ignore_index masking over
the padded loss), so it does not over-mark. INT8 quantization is lossless (2.30% DER,
2.5 MB).test.txt, though ~1.7% of test sentences also appear
in train via near-duplicate aggregation — a smaller exposure than typical external
baselines on the same set.Embedding(236, 128) → 2-layer bidirectional LSTM(256) → Linear(512, 75).
Normalization is NFD with symbols (Unicode So) dropped; the model predicts one of
75 NFD-based diacritic classes per base character (it also restores hamzas and the
superscript alef).| file | what |
|---|---|
rawi_v2.onnx | fp32 ONNX, dynamic sequence length |
rawi_v2.int8.onnx | INT8 dynamic quantization (2.5 MB, lossless) |
rawi_v2.vocab.json | char_to_idx / diac_to_idx |
diacritization_model_lstm_2.pth | original PyTorch checkpoint |
rawi_v2_train.ipynb | training notebook |
1from text2tashkeel import Diacritizer # pip install text2tashkeel
2Diacritizer("rawi-v2").diacritize("بسم الله الرحمن الرحيم")
3# بِسْمِ اللَّهِ الرَّحْمَنِ الرَّحِيمِtext2tashkeel bundles this model and reproduces the exact NFD normalization and
letter-only decode in pure Python (no PyTorch at inference).