BiLSTM-CRF model for splitting concatenated strings into words. Trained on millions of domain names, brand names, personal names, and multilingual phrases.
1import dksplit
2
3dksplit.split("chatgptlogin")
4# ['chatgpt', 'login']
5
6dksplit.split("spotifywrapped")
7# ['spotify', 'wrapped']
8
9dksplit.split("mercibeaucoup")
10# ['merci', 'beaucoup']
11
12dksplit.split_batch(["openaikey", "microsoftoffice", "bitcoinprice"])
13# [['openai', 'key'], ['microsoft', 'office'], ['bitcoin', 'price']]
14
15# Top-k candidates, best first
16dksplit.split3("noranite")
17# [['nora', 'nite'], ['noranite'], ['nor', 'anite']]
18
19dksplit.split5("pikahug")
20# [['pikahug'], ['pika', 'hug'], ['pik', 'ahug'], ['pikah', 'ug'], ['pi', 'kahug']]
21
22dksplit.split_topk("chatgptlogin", k=3)
23# [['chatgpt', 'login'], ['chatgptlogin'], ['chatgpt', 'log', 'in']]
1,000 hand-audited domain prefixes drawn from the
Newly Registered Domains Database (NRDS) (.com feed). No filtering or cherry-picking on segmentation difficulty. Ground truth was established through multi-model cross-validation (BiLSTM, Qwen 9B LoRA, Gemma 31B) and human audit. Each row provides a primary
truth and an optional
might_right field for genuinely ambiguous cases.
Both benchmark sets ship in the GitHub repo's
/benchmark
directory and on Hugging Face as
ABTdomain/dksplit-benchmark.
To explore domain data yourself, register at
domainkits.com — fresh .com NRD downloads are free.
The model outputs emission scores. CRF decoding is done separately using the parameters in dksplit.npz.
1import numpy as np
2import onnxruntime as ort
3
4# Load model
5sess = ort.InferenceSession("dksplit-int8.onnx")
6crf = np.load("dksplit.npz")
7
8# Encode input
9CHAR_MAP = {c: i+2 for i, c in enumerate("abcdefghijklmnopqrstuvwxyz0123456789")}
10text = "chatgptlogin"
11ids = np.array([[CHAR_MAP.get(c, 1) for c in text]], dtype=np.int64)
12
13# Get emissions
14emissions = sess.run(["emissions"], {"chars": ids})[0]
15
16# CRF Viterbi decode
17trans = crf["transitions"]
18start_t = crf["start_transitions"]
19end_t = crf["end_transitions"]
20
21score = start_t + emissions[0, 0]
22history = []
23for t in range(1, emissions.shape[1]):
24 ns = score[:, None] + trans + emissions[0, t, None, :]
25 history.append(np.argmax(ns, axis=0))
26 score = np.max(ns, axis=0)
27best = [np.argmax(score + end_t)]
28for h in reversed(history):
29 best.append(h[best[-1]])
30best.reverse()
31
32# Decode to words
33words, cur = [], []
34for ch, lb in zip(text, best):
35 if lb == 1 and cur:
36 words.append("".join(cur))
37 cur = [ch]
38 else:
39 cur.append(ch)
40if cur:
41 words.append("".join(cur))
42print(words) # ['chatgpt', 'login']
The model was trained on the Leonardo Booster supercomputer at CINECA, Italy, with computing resources provided by the
EuroHPC Joint Undertaking through the Playground Access program (EHPC-AIF-2026PG01-281). We thank EuroHPC JU for enabling SMEs to explore new possibilities with world-class HPC infrastructure.
CC BY 4.0. Attribution required: credit "DKSplit by
ABTdomain" in your README, documentation, about page, or API response metadata.
