Wen Reader — On-Device Chinese NLP Models
Model weights for
Wen Reader, an open-source Chinese EPUB reader for iOS with popup dictionary.
These models run on-device via CoreML to provide context-aware Chinese word segmentation and word sense disambiguation.
Models
cws-span-scorer-electra-base
Chinese Word Segmentation model using span scoring.
- Architecture: ELECTRA-base (12 layers, 768 hidden, 12 heads) + span scoring MLP head
- Base model: hfl/chinese-electra-180g-base-discriminator
- Parameters: ~102M (encoder) + span head
- Task: Given a sentence, score candidate word spans to find optimal segmentation
- Eval: 90/97 perfect segmentation on hand-curated test set (93/97 including acceptable over-splits)
cws-span-scorer-electra-small
Smaller/faster variant of the above for lower-end devices.
- Architecture: ELECTRA-small (12 layers, 256 hidden, 4 heads) + span scoring MLP head
- Base model: hfl/chinese-electra-180g-small-discriminator
- Parameters: ~12M (encoder) + span head
- Eval: 85/97 perfect (91/97 including acceptable over-splits)
wsd-biencoder-gte-base
Word Sense Disambiguation bi-encoder for selecting the correct dictionary definition in context.
- Architecture: GTE-base-zh fine-tuned as a bi-encoder (SentenceTransformers)
- Base model: thenlper/gte-base-zh
- Parameters: ~102M
- Task: Encode context sentence and candidate sense labels, rank by cosine similarity
- Eval: Top-1 accuracy 88.7%, Top-3 99.6%, MRR 0.936 on 239 hand-curated disambiguation examples
Usage
These models are used by the Wen Reader iOS app via CoreML conversion.
To build the app from a fresh clone:
1cd ml
2
3# 1. Download model weights from HF Hub
4uv run python scripts/download_models.py
5
6# 2. Export to CoreML
7uv run python scripts/export_coreml.py span --model-dir models/cws_span_scorer_electra_base/final
8uv run python scripts/export_coreml.py wsd --model-dir models/wsd_biencoder_gte_base/final
9
10# 3. Bundle into app resources (vocab, CoreML packages, CEDICT database)
11./scripts/run_pipeline.sh bundle
Or use the pipeline script:
1./scripts/run_pipeline.sh download
2./scripts/run_pipeline.sh export
3./scripts/run_pipeline.sh bundle
Training
CWS Span Scorer
Fine-tuned on a custom dataset built from:
- ICWB2 segmentation corpus (MSR, auto-matched where greedy segmenter agrees with gold, plus LLM-annotated disagreement cases)
- Chinese Wikipedia and OpenSubtitles sentences, LLM-annotated (Claude Opus)
Sentences with ambiguous segmentation boundaries (where multiple valid CEDICT words overlap) are
identified, then an LLM annotates the correct segmentation. The model scores all candidate word
spans at each ambiguous position using a span scoring MLP head (taking span boundary token
representations + learned width embeddings as input). A dynamic programming decoder finds the
optimal segmentation at inference time. Training uses cross-entropy loss over candidate spans at
each ambiguous position. Encoder and head use discriminative learning rates.
WSD Bi-encoder
Fine-tuned on CC-CEDICT sense clusters with data from:
- MiCLS WSD corpus (mapped to CEDICT senses via BGE embedding similarity)
- LLM-generated context examples for each sense cluster (Claude Opus and Sonnet)
- LLM-annotated ebook sentences segmented by the CWS span scorer (Claude Sonnet)
Sense clusters are derived from CC-CEDICT entries with related senses merged using LLM-assisted
clustering. Chinese sense labels are LLM-translated from the English CEDICT definitions.
Training uses grouped cross-entropy loss: contexts and sense labels are encoded by the same model,
cosine similarity is computed between context embeddings and sense embeddings, and the loss is
cross-entropy over the similarity scores (with temperature scaling at τ=0.1). Contexts are grouped
by word in each batch so sense embeddings are encoded once and reused across all contexts for that word.
