Views
No views yet
| Property | Value |
|---|---|
| Language | Kabyle (kab) |
| Type | Sentence tokenizer |
| Architecture | CNN + BiLSTM |
| Training data | Tatoeba Kabyle sentences (~789K sentences) |
| Dev F1 | 99.19% |
| Token F1 | 99.96% |
| Sentence F1 | 98.43% |
| ONNX size | 0.62 MB |
| Vocab size | 223 characters |
kab.onnx — ONNX runtime modelkab.pt — PyTorch checkpointvocab.json — Character vocabulary (223 entries)config.json — Model hyperparameterstokenizer_config.json — HF tokenizer config1import stanza
2nlp = stanza.Pipeline(
3 lang="kab",
4 processors="tokenize",
5 tokenize_model_path="kab.pt"
6)
7doc = nlp("Amcic ha-t-an deg uxxam-nneɣ. Teciḍ fell-as?")
8for sent in doc.sentences:
9 print(sent.text)
10# Amcic ha-t-an deg uxxam-nneɣ.
11# Teciḍ fell-as?1import onnxruntime as ort
2import numpy as np
3
4sess = ort.InferenceSession("kab_tokenizer.onnx")
5# units: (batch, seq_len) int64 — char IDs from vocab.json
6# features: (batch, seq_len, 5) float32 — Stanza features
7units = np.zeros((1, 100), dtype=np.int64)
8features = np.zeros((1, 100, 5), dtype=np.float32)
9outputs = sess.run(None, {"units": units, "features": features})
10# outputs[0] shape: (batch, seq_len, 3) — logits for B/I/O| Input | Tokens |
|---|---|
Ad tseddumt ɣer Taskriwt. | ['Ad', 'tseddumt', 'ɣer', 'Taskriwt', '.'] |
Aweḍ ɣer Tezmalt. | ['Aweḍ', 'ɣer', 'Tezmalt', '.'] |
Tettawḍem ɣer Kendira. | ['Tettawḍem', 'ɣer', 'Kendira', '.'] |
Efk-asen tizwal-nni. | ['Efk-asen', 'tizwal-nni', '.'] |
Melmi ara ad d-taɣeḍ lmitra? | ['Melmi', 'ara', 'ad', 'd-taɣeḍ', 'lmitra', '?'] |
ɛ / Ɛ (open e)ɣ / Ɣ (voiced velar fricative)ṭ, ḍ, č, ǧ (emphatic and palatal consonants)