Fon - Wikilangs Models

Comprehensive Research Report & Full Ablation Study

📋 Repository Contents

Models & Assets

• Tokenizers (8k, 16k, 32k, 64k)
• N-gram models (2, 3, 4, 5-gram)
• Markov chains (context of 1, 2, 3, 4 and 5)
• Subword N-gram and Markov chains
• Embeddings in various sizes and dimensions (aligned and unaligned)
• Language Vocabulary
• Language Statistics

Analysis and Evaluation

• 1. Tokenizer Evaluation
• 2. N-gram Model Evaluation
• 3. Markov Chain Evaluation
• 4. Vocabulary Analysis
• 5. Word Embeddings Evaluation
• 6. Morphological Analysis (Experimental)
• 7. Summary & Recommendations
• Metrics Glossary
• Visualizations Index

---

1. Tokenizer Evaluation

Results

Tokenization Examples

Key Findings

• Best Compression: 64k achieves 4.124x compression
• Lowest UNK Rate: 8k with 0.1627% unknown tokens
• Trade-off: Larger vocabularies improve compression but increase model size
• Recommendation: 32k vocabulary provides optimal balance for production use

---

2. N-gram Model Evaluation

Results

Top 5 N-grams by Size

Key Findings

• Best Perplexity: 2-gram (subword) with 265
• Entropy Trend: Decreases with larger n-grams (more predictable)
• Coverage: Top-1000 patterns cover ~45% of corpus
• Recommendation: 4-gram or 5-gram for best predictive performance

---

3. Markov Chain Evaluation

Results

Generated Text Samples (Word-based)

1. tɔn mɛ wli hwe ɔ huzu tokpɔnlavi agɔnkanmɛ tɔn bo ɖyɔ ɛ ylɔ ɛ ɖo yovogbè
2. ɖo tokpɔn alibori e nɔ̀ kpénukún tovixixa wǔ é kpo hɛnnu mɛ bo nɔ nyì do
3. e ɖo lé e é mɛ xwédo 1 lɛ nukɔnnɔtɔ hwɛxo tɔn ayi e yovo hwan

1. tɔn mɛ ɖò totaligbé gbadahweji benɛɛtò tɔn lɛ mi na mɔ xogbè to ɔ tɔn ɖo tantɔn
2. mɛ ɖo benɛ ayi e jì é ɖěè lěè akpɔkpɔ ɖé ɖe ɔ è sɔ ɛ ɖɛmɛnu
3. mɛ e lɛ́zun gletoxo do sɛ̀nxwĭ jí sin azan ayizin 6 xwejisùn léxwé tɔn mɛ toxoɖɔgbɛ tɔn

1. tɔn mɛ ɖo atacora e lɛ́ nyi gletoxo do sɛ̀nxwĭ jí sin azan ayizin 6 xwejisùn lé xwélé
2. jì é ɖěè zinvie ɖo tokpɔnlavi zinvié tɔn mɛ ɖo benɛɛto mɛ bo nyi sɔmi sɔmi ɖɛ̌mɛnu lɛ
3. ayi e jì é ɖěè tokpɔnlávì tayaku tɔn ɔ nyi tokpɔnlavi ɖokpo ɖo wò 10 ě ɖo tokpɔnla

1. ayi e jì é ɖěè dovogon ɖo tokpɔnlavi zogbodomey tɔn mɛ ɖo zou e lɛ́ nyí gletoxo ɖò sɛ̀nxwí
2. e jì é ɖěè bouhanrou ɖo tokpɔnlavi gomparou tɔn mɛ ɖo alibori e lɛ́ nyi gletoxo ɖo sɛ̀nxwĭ jí
3. e ɖěè mɛ e axɔsuxwe insae instad e nɔ̀n kpé nunkún tovixixa wǔ é lɛn xɔta 248 nǔ gbɛtɔ

Generated Text Samples (Subword-based)

1. _be,._ɖo_ɖěè_e"_
2. nɔn_kuɖoudoku_to
3. o_é_mbe_gblɛn_ɖò

1. n_kan)_xɔtan_è_ɖo
2. o_tɔnla_akanɖie_ɖ
3. _tokpé_dodo_tɛntr

1. ɔn_atlant_dolore_t
2. tɔn_ɖó_azinkpo_ɔ,_
3. _tɔn_ɔ_tɔn_léxwé_d

1. _tɔn._ɖo_tokpɔn_atu
2. tɔn_lɛ_sin_azǎn_20ɔ
3. _ɖo_tokpɔnlavi_tɔn,

Key Findings

• Best Predictability: Context-4 (word) with 95.3% predictability
• Branching Factor: Decreases with context size (more deterministic)
• Memory Trade-off: Larger contexts require more storage (98,034 contexts)
• Recommendation: Context-3 or Context-4 for text generation

---

4. Vocabulary Analysis

Statistics

Most Common Words

Least Common Words (from vocabulary)

Zipf's Law Analysis

Coverage Analysis

Key Findings

• Zipf Compliance: R²=0.9939 indicates excellent adherence to Zipf's law
• High Frequency Dominance: Top 100 words cover 63.7% of corpus
• Long Tail: 1,148 words needed for remaining 0.6% coverage

---

5. Word Embeddings Evaluation

5.1 Cross-Lingual Alignment

5.2 Model Comparison

Key Findings

• Best Isotropy: mono_32d with 0.6254 (more uniform distribution)
• Semantic Density: Average pairwise similarity of 0.3821. Lower values indicate better semantic separation.
• Alignment Quality: Aligned models achieve up to 5.2% R@1 in cross-lingual retrieval.
• Recommendation: 128d aligned for best cross-lingual performance

---

6. Morphological Analysis (Experimental)

6.1 Productivity & Complexity

6.2 Affix Inventory (Productive Units)

Productive Prefixes

Productive Suffixes

6.3 Bound Stems (Lexical Roots)

6.4 Affix Compatibility (Co-occurrence)

6.5 Recursive Morpheme Segmentation

6.6 Linguistic Interpretation

Automated Insight: The language Fon shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.

Note on Idiomaticity: The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.

---

7. Summary & Recommendations

Production Recommendations

---

Appendix: Metrics Glossary & Interpretation Guide

Tokenizer Metrics

Definition: The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.

Intuition: Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.

What to seek: Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.

Definition: Mean number of characters per token produced by the tokenizer.

Intuition: Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.

What to seek: Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.

Definition: Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.

Intuition: Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.

What to seek: Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.

N-gram Model Metrics

Definition: Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.

Intuition: If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.

What to seek: Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.

Definition: Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.

Intuition: High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.

What to seek: Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.

Definition: Percentage of corpus occurrences explained by the top K most frequent n-grams.

Intuition: High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.

What to seek: Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.

Markov Chain Metrics

Definition: Mean entropy across all contexts, measuring average uncertainty in next-word prediction.

Intuition: Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).

What to seek: Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.

Definition: Average number of unique next tokens observed for each context.

Intuition: High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).

What to seek: Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.

Definition: Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.

Intuition: 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.

What to seek: Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.

Vocabulary & Zipf's Law Metrics

Definition: The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.

Intuition: A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.

What to seek: Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.

Definition: Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.

Intuition: R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.

What to seek: R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.

Definition: Cumulative percentage of corpus tokens accounted for by the top N words.

Intuition: Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.

What to seek: Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.

Word Embedding Metrics

Definition: Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.

Intuition: High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.

What to seek: Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.

Definition: Mean magnitude (L2 norm) of word vectors in the embedding space.

Intuition: Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.

What to seek: Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).

Definition: Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).

Intuition: Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.

What to seek: Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.

Definition: t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.

Intuition: Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.

What to seek: Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.

General Interpretation Guidelines

1. Compare within model families: Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
2. Consider trade-offs: Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
3. Context matters: Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
4. Corpus influence: All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
5. Language-specific patterns: Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.

Visualizations Index

---

About This Project

Data Source

Project

Maintainer

Citation

License

Links

• 🌐 Website: wikilangs.org
• 🤗 Models: huggingface.co/wikilangs
• 📊 Data: wikipedia-monthly
• 👤 Author: Omar Kamali
• 🤝 Sponsor: Featherless AI

---