Phazel/

fa_ent_news_md

Persian named entity recognizer optimized for CPU, with its own internal tok2vec. Labels: PER, LOC, ORG, DAT, MON, TIM, PCT.

Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. The ner component is trained on the NER layer shipped in UD_Persian-PerDT's not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. Those labels are SILVER: the treebank README states they were produced by the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in performance.ents_per_type should be read before relying on them. This is the md tier: identical architecture to the sm pipeline plus static floret vectors (50,000 rows x 300 dimensions, minn=maxn=5, hash_count=2) trained on 400,000 Persian documents. floret hashes subwords into a fixed table, so there are no out-of-vocabulary tokens and token.has_vector is always True. That matters for Persian, where inconsistent ZWNJ (U+200C) usage splits one word across several surface forms (mi-ravad written joined, with ZWNJ, or with a space) that a classic word-vector table would miss.