File	Description
`crf_gpu.safetensors`	Model weights + labels metadata (GPU inference)
`crf_bidir.pkl`	Trained sklearn-crfsuite model (CPU inference)
`crf_forward.pkl`	Causal (left-context only) variant
`crf_gpu.py`	GPU inference module (`CRFGPUModel`)
`ldgc/vocab.py`	Character vocabulary (required by crf_gpu.py)

Quick start — GPU inference

python

from crf_gpu import CRFGPUModel

model = CRFGPUModel.from_pretrained("crf_gpu.safetensors", device="cuda")
preds = model.predict(["turkce cok guzel", "bugun hava guzel"], batch_size=512)
# → ["türkçe çok güzel", "bugün hava güzel"]

Quick start — CPU inference (sklearn-crfsuite)

python

import pickle
crf  = pickle.load(open("crf_bidir.pkl", "rb"))
sent = "turkce cok guzel"
feats = [...]   # see crf_gpu.py encode_batch for feature extraction
pred  = "".join(crf.predict([feats])[0])

Streaming billions of sentences (GPU)

python

from crf_gpu import predict_stream

with open("large_corpus.txt") as f:
    for restored in predict_stream(model, f, batch_size=512):
        process(restored)

Label vocabulary

37 output labels:

' ' '!' ',' '.' '?' 'a' 'b' 'c' 'd' 'e' 'f' 'g' 'h' 'i' 'j' 'k' 'l' 'm' 'n' 'o' 'p' 'r' 's' 't' 'u' 'v' 'y' 'z' 'â' 'ç' 'î' 'ö' 'û' 'ü' 'ğ' 'ı' 'ş'

Feature set (bidirectional)

Unigram: current char, ±1 char, ±2 char
Bigram: (−1,0), (−2,−1), (0,+1), (+1,+2) character pairs
Boolean: is_vowel, is_diac_src, −1/+1:is_vowel, BOS, EOS
Categorical: word_pos ∈ {start, mid, end}
Bias

Citation

bibtex

@misc{ldgc2025,
  title  = {LDGC: Latent Dynamics Grammar Corrector for Turkish},
  author = {Emircan EROL},
  year   = {2025},
  url    = {https://github.com/emircan-erol/tr-grammar},
}

emircanerol/turkish-diacritic-crf

Turkish Diacritic Restoration — Bidirectional CRF

Performance (medium corruption: p_strip=0.70, p_drop=0.15, p_typo=0.05)

Files

Quick start — GPU inference

Quick start — CPU inference (sklearn-crfsuite)

Streaming billions of sentences (GPU)

Label vocabulary

Feature set (bidirectional)

Citation

emircanerol/
turkish-diacritic-crf