en-or-not: English vs non-English text detection
en-or-not is a small byte-level convolutional neural network that predicts
whether an input is English (EN) or not English (NOT-EN). It is intended
for short-text filtering, not for identifying which non-English language is
present.
The implementation and training/provenance companion for v1 is on
GitHub.
Model
- Architecture:
TinyByteCNN_EN, six convolutional blocks
- Parameters: 156,642
- Input: raw UTF-8 bytes, truncated to 256 bytes and zero-padded
- Output: two logits in the order
NOT-EN, EN
- Files:
model.onnx for ONNX Runtime and model.safetensors for PyTorch
Published OOD evaluation
The primary evaluation set is
evaluation/ood_set.jsonl,
with its provenance and selection rules in
evaluation/ood_manifest.json. It contains
17,544 real Tatoeba sentences:
- 8,772 English examples
- 8,772 non-English examples
- 52 non-English language strata, covering multiple scripts and language
families
- No synthetic fallback rows
- No duplicate normalized-text hashes
- Every row is valid for the model's 256-byte input limit
The set was built from the pinned Tatoeba sentence export dated 2026-07-18.
Tatoeba data is released under CC-BY 2.0. Each row retains its Tatoeba source
ID and attribution URL. The benchmark is source-level OOD relative to the
training sources documented in the linked project: Project Gutenberg,
papluca/language-identification, and optional OSCAR. This is not a guarantee
of zero lexical overlap with undisclosed training artifacts.
ONNX Runtime result:
1accuracy: 17007/17544 = 96.9391%
295% Wald CI: 96.6842%-97.1940%
3EN: 8465/8772 = 96.5002%
4NOT-EN: 8542/8772 = 97.3780%
5language macro accuracy: 97.3802% across 52 non-English languages
The benchmark is balanced by class, so the overall accuracy is also balanced
accuracy. Short examples are harder than longer examples in this set:
1short: 6931/7327 = 94.5953%
2medium: 9390/9531 = 98.5206%
3long: 686/686 = 100.0000%
Rebuild and evaluate it with:
1python3 scripts/build_ood_set.py \
2 --source /path/to/sentences.tar.bz2 \
3 --output evaluation/ood_set.jsonl \
4 --manifest evaluation/ood_manifest.json
5
6python3 scripts/evaluate_onnx.py evaluation/ood_set.jsonl
The old hand-authored evaluation/hostile_ood.jsonl file is retained as an
adversarial diagnostic set. It is not included in the published OOD figure.
Quick start
1import numpy as np
2import onnxruntime as ort
3
4session = ort.InferenceSession("model.onnx")
5
6def english_probability(text: str) -> float:
7 data = text.encode("utf-8", errors="ignore")[:256]
8 inputs = np.zeros((1, 256), dtype=np.int64)
9 inputs[0, :len(data)] = list(data)
10 logits = session.run(["logits"], {"input_bytes": inputs})[0][0]
11 probabilities = np.exp(logits - np.max(logits))
12 probabilities /= probabilities.sum()
13 return float(probabilities[1])
14
15print(english_probability("Hello world!"))
Training result and limitations
The original model metadata reports 99.94% validation accuracy on 17,543
training-pipeline samples. The source data, split, and training evaluator are
not included here, so that number is a reported training result, not an
independently reproducible general-accuracy claim.
This is binary classification, not language identification. Very short text,
capitalization, transliteration, mixed language, code, URLs, numbers, and
non-linguistic strings can be out of distribution. Confidence scores are model
probabilities, not calibrated guarantees. Evaluate on data representative of
the deployment domain before using the model as a gate or safety-critical
decision.
License
The model artifacts and code are released under the MIT License. The published
evaluation text is sourced from Tatoeba and remains subject to CC-BY 2.0
attribution requirements.
