You can deploy this model and start earning money today!

TigreGotico/
glotlid-onnx

Name: glotlid-onnx
Availability: InStock

onnx

language-identification

Views

No views yet

GlotLID v3 - ONNX

ONNX export of cis-lmu/glotlid model_v3.bin, a fastText supervised language identifier that covers 2102 labels in the form iso639-3_Script (for example eng_Latn, por_Latn, glg_Latn).

The licence is Apache-2.0, inherited from GlotLID. Please cite the GlotLID authors if you use this model.

What the graph does and does not do

fastText inference has five steps. The ONNX graph holds only the last two:

Step	Where
1. Tokenize the text	Python (`glotlid_hash.py`)
2. Character n-grams per word (`minn`..`maxn`)	Python
3. Hash the n-grams into buckets	Python
4. Average the embedding rows of all feature ids	ONNX
5. Multiply by the output matrix, then softmax	ONNX

Steps 1 to 3 are string processing. ONNX has no portable operator for fastText's FNV-1a byte hash over UTF-8, so that work stays in Python. The split keeps the graph a pure numeric pipeline:

input_ids -> Gather(input_matrix) -> ReduceMean(axis=0) -> MatMul(output_matrix) -> Softmax -> probs

glotlid_hash.py is the reference implementation of steps 1 to 3. It reproduces fastText's Dictionary::getLine, Dictionary::computeSubwords and Dictionary::hash. Two details are easy to get wrong:

fastText casts each byte to a signed int8_t before the FNV-1a XOR, so bytes >= 0x80 are sign-extended. Without this, every non-ASCII n-gram lands in the wrong bucket.
Each line ends with the </s> end-of-sentence token, and that token contributes its own vocabulary row.

Files

File	Size	Purpose
`glotlid.onnx`	1.68 GB	fp32 graph
`glotlid.int8.onnx`	419 MB	dynamic int8 graph
`labels.json`	44 KB	2102 labels, in output order
`vocab.txt`	5.6 MB	634361 vocabulary words, in id order
`config.json`	105 B	`dim`, `minn`, `maxn`, `bucket`, `nwords`, `nlabels`
`glotlid_hash.py`	4.8 KB	reference feature extractor

Model arguments

dim=256  minn=2  maxn=5  bucket=1000000  wordNgrams=1
nwords=634361  nlabels=2102
input_matrix=(1634361, 256)   # nwords + bucket
output_matrix=(2102, 256)

Tensors

Name	Direction	Type	Shape
`input_ids`	input	int64	`[num_features]`
`probs`	output	float32	`[2102]`

probs is indexed by the order of labels.json.

Usage

python

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download

from glotlid_hash import GlotLIDFeaturizer

d = snapshot_download("TigreGotico/glotlid-onnx")
feat = GlotLIDFeaturizer.from_files(f"{d}/vocab.txt", f"{d}/config.json")
labels = json.load(open(f"{d}/labels.json", encoding="utf-8"))
sess = ort.InferenceSession(f"{d}/glotlid.onnx", providers=["CPUExecutionProvider"])

def detect(text, k=5):
    probs = sess.run(None, {"input_ids": feat(text)})[0]
    top = np.argsort(-probs)[:k]
    return [(labels[i], float(probs[i])) for i in top]

print(detect("O tempo está moi bo hoxe en Santiago"))
# [('__label__glg_Latn', 0.98...), ...]

Parity with fastText

78 short samples across 60 languages, chosen for script and resource diversity: Portuguese, Galician, Catalan, Spanish, Basque, English, French, German, Italian, Dutch, Arabic, Chinese, Japanese, Korean, Hindi, Russian, Ukrainian, Greek, Swahili, Turkish, Polish, Czech, Finnish, Hungarian, Romanian, Swedish, Hebrew, Persian, Thai, Vietnamese, Indonesian, Tagalog, Amharic, Hausa, Yoruba, Igbo, Zulu, Somali, Bengali, Tamil, Telugu, Malayalam, Nepali, Georgian, Armenian, Icelandic, Welsh, Irish, Maltese, Esperanto, Quechua, Guarani, Haitian Creole, Malagasy, Kinyarwanda, Mongolian, Khmer, Lao, Burmese, Sinhala.

Each sample was compared against fasttext_model.predict(text, k=5).

Model	Top-1 agreement	Top-5 identical order	Mean abs. prob delta (top-1)	Max
`glotlid.onnx` (fp32)	100.00 % (78/78)	83.3 %	1.03e-05	1.39e-05
`glotlid.int8.onnx`	100.00 % (78/78)	67.9 %	6.08e-04	1.81e-02

The fp32 probability delta of about 1e-05 is not export error. fastText returns probabilities through a 512-entry logarithm lookup table whose entries carry a +1e-5 term, so its reported values sit about 1e-5 above the true softmax. The ONNX graph returns the exact softmax. This also explains the top-5 order differences: they occur only among labels whose probabilities are below the resolution of that table, where fastText's own ranking is arbitrary.

The int8 graph keeps every top-1 decision but its probability values move by up to 0.018, so use fp32 if you threshold on confidence.

Reproduction

glotlid_hash.py was validated against fastText's own get_subwords() on 2006 tokens (2000 sampled from the vocabulary plus out-of-vocabulary strings in Greek, Devanagari, Han, Latin-extended and Inuktitut syllabics): 0 mismatches in the full feature-id lists.