Three adapters that lost. This repository exists so that a negative result stays
reproducible instead of becoming folklore. None of these is a shipping model; the models
that shipped are in
digitable-lol/digit-router-0.6b
and digitable-lol/digit-router-1.7b.
1. Read this before you read any number
1.1 These are routers, not assistants
Each adapter turns its base model into a two-step tool router for the Digit verified
agent: step 1 maps a query to one of 14 tool categories, step 2 emits a tool call with
extracted arguments given that category's JSON schemas — or refuses. The model never
produces the content of an answer; that comes from a deterministic utility, a verbatim
corpus quote, or a formal certificate.
This framing is the whole reason the Russian-SFT bases lose. Instruction tuning on Russian
dialogue improves free-form generation — the one capability this architecture deliberately
does not use.
1.2 The headline result: Russian SFT without a rebuilt tokeniser buys nothing
Every Vikhr model measured here gives exactly zero fertility gain over its own Qwen
base — not "a small gain", zero: the numbers are identical to four decimal places,
because Vikhr is a Russian SFT over Qwen with an unchanged vocabulary. The same holds
for QVikhr-3 over Qwen3. The expectation "Russian model → Russian tokeniser" simply does
not hold, and it has to be checked by measurement rather than by the name.
The only base that gains is ruadapt, which actually extends the vocabulary — and it pays
for the gain in argument accuracy. Details in § 2 and § 3.
1.3 A counted refusal is not a conscious refusal
The eval harness scores an empty or unparseable answer as a refusal, so a model that
simply breaks scores like a model that knows when to decline. The refusal column below is
the conscious one: the router emitted an explicit {"refuse": "..."} object. The
unreadable column is what the counted number would have absorbed. For reference, the
untuned Qwen3-0.6B base scores 75.3 % counted against 9.3 % conscious, with 115
unreadable answers out of 250 — a 66 pp gap that is entirely broken output.
1.4 Single seed, dataset v1, and one candidate pair is missing
Every adapter here was trained once, with seed=20260802, on dataset v1
(deliberately: v1 isolates the effect of the base from the effect of the data). Differences
of 1–2 pp on 250 tasks are 2–5 tasks and are not distinguishable from initialisation noise.
Two candidates in the original shortlist were never trained:
Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct (same tokeniser and same nature as the Vikhr
models that were tested) and Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct (three times worse
fertility than ruadapt). The argument against both is indirect, and it is stated here
rather than hidden: strictly speaking they are two untested candidates.
2. Measurement 1 — tokeniser fertility
The only number about a base model obtainable without training anything. It answers two
practical questions at once: how much work a request costs (a token is the unit of work)
and how many tool schemas fit in the context budget.
Corpus: three slices of real data, not "typical Russian text" — 3 000 paragraphs of Russian
course prose (941 376 characters, 136 722 words), 4 000 real user queries (186 289
characters), and the 14 step-2 system prompts (19 640 characters) that determine how many
tools fit in context.
Model
Vocab
RU prose, tok/word
chars/tok
Queries, tok/word
Step-2 schemas, total tokens
median schema
Qwen/Qwen3-0.6B (control)
151 669
2.344
2.94
3.056
6 486
487
Qwen/Qwen3-1.7B
151 669
2.344
2.94
3.056
6 486
487
Qwen/Qwen2.5-0.5B-Instruct
151 665
2.344
2.94
3.056
6 486
487
Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct
151 665
2.344
2.94
3.056
6 486
487
Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct
151 665
2.344
2.94
3.056
6 486
487
Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning
151 669
2.344
2.94
3.056
6 486
487
RefalMachine/RuadaptQwen2.5-1.5B-instruct
145 152
1.686 (−28.1 %)
4.08
2.597 (−15.0 %)
5 394 (−16.8 %)
407
Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct
128 256
2.177 (−7.1 %)
3.16
2.808 (−8.1 %)
6 258 (−3.5 %)
466
Look at the six identical rows. Four different "Russian" models and two Qwen controls
produce byte-identical fertility, because none of them touched the vocabulary.
Only ruadapt gains, and it gains a lot: 1.686 tokens per word against 2.344 on
Russian prose — 28 % fewer — with a smaller vocabulary (145 152 against 151 669). On our
queries the gap is narrower (−15 %) because queries are short and full of Latin script,
numbers and identifiers, where there is nothing to gain. In context terms, all 14 step-2
schemas take 5 394 tokens instead of 6 486, so roughly 20 % more schemas fit the same
budget.
The figure is confirmed twice on real training: ruadapt's median training example is 447
tokens against 562 for Qwen3 (p99 709 against 811), and one epoch over the same dataset
passes 14.85 M tokens against 18.21 M — 18.5 % fewer — in 56.4 minutes against 71.8.
Source: train/RESULTS.md § 2, raw numbers in train/fertility/fertility.json.
3. Measurement 2 — quality, identical LoRA on five bases
Everything identical except the base model: same dataset (v1), same LoRA (r=32,
alpha=64, dropout 0.05, the same seven target modules), lr=1e-4 cosine with 3 % warmup,
effective batch 16 (4 × accumulation 4), max_length=1024, adamw_bnb_8bit, one epoch,
seed=20260802, greedy decoding, the same two-step inference. All 250 tasks, no sampling,
unmodified eval/scoring.py.
Base
Params
tool_accuracy
arg_accuracy
conscious refusal (red-team)
false_answer_rate
over_refusal
unreadable
training
Qwen3-0.6B (incumbent)
0.6B
83.0 %
93.7 %
75.3 %
17.2 %
12.0 %
7
35.7 min
Vikhr-Qwen-2.5-0.5b ← in this repo
0.5B
81.0 %
83.8 %
63.3 %
27.2 %
11.0 %
11
24.7 min
RuadaptQwen2.5-1.5B ← in this repo
1.5B
88.0 %
85.7 %
84.0 %
14.8 %
9.0 %
4
56.4 min
QVikhr-3-1.7B ← in this repo
1.7B
84.0 %
90.1 %
82.7 %
15.6 %
11.0 %
3
72.2 min
Qwen3-1.7B (winner)
1.7B
86.0 %
92.7 %
83.3 %
12.8 %
9.0 %
4
71.8 min
On the two gaps the whole exercise was about:
Base
false_premise: answered with a tool call
missing_param: answered with a tool call
Qwen3-0.6B
13/30
16/40
Vikhr-Qwen-2.5-0.5b
13/30
22/40
RuadaptQwen2.5-1.5B
9/30
12/40
QVikhr-3-1.7B
11/30
13/40
Qwen3-1.7B
6/30
15/40
3.1 Russian SFT does not help — it hurts, and it shows on two honest pairs
Vikhr-Qwen-2.5-0.5b vs Qwen3-0.6B (roughly one size): worse on everything —
routing 81 vs 83, arguments 83.8 vs 93.7, conscious refusal 63.3 vs 75.3,
false_answer_rate27.2 vs 17.2, decoys 22/40 vs 16/40. On the main diagnosed gap
it is markedly worse. This is the direct answer to the original proposal "take Vikhr on
a Qwen2.5-0.5B base": the measured answer is don't.
QVikhr-3-1.7B vs Qwen3-1.7B — the strong pair, because this is the same base
model, the same tokeniser and the same size, differing only by a Russian SFT layer on
top. Worse on everything that matters: false_answer_rate 15.6 vs 12.8, arguments
90.1 vs 92.7, false premises 11/30 vs 6/30. The Russian SFT added nothing and blurred
part of the original capability.
The cause follows from § 1.1: in a verified architecture the model classifies and extracts,
it does not write. Instruction tuning on Russian dialogue improves what we do not use.
3.2 An extended Russian tokeniser is the only thing that really works — and it is not free
ruadapt takes the best routing (88.0 %), the best conscious refusal (84.0 %) and the best
decoy result (12/40) — and the worst argument accuracy among the large models:
85.7 % against 92.7 %.
This is not an artefact of ruadapt attempting more tasks. On the 84 tasks both models
attempted, Qwen3-0.6B gets 85.7 % of arguments right and ruadapt 79.8 %. Of ruadapt's
12 wrong-argument tasks, 10 are secondary parameters (hashFunction, cypherAlgo,
font, paragraphs, indentSize, timeSpan, defaultCountryCode) and only 2 are the
main literal (inputMarkdown, rawPhone). The model confidently identifies which tool
is needed and fills its settings worse.
Choosing between Qwen3-1.7B and ruadapt was choosing between 6 argument tasks and
2 routing tasks. In a verified architecture a tool called with a wrong argument returns a
verified wrong answer; a wrongly chosen tool usually fails its own validation and is
visible. So argument accuracy weighs more — and it agrees with the harness's primary
metric, false_answer_rate, on which Qwen3-1.7B wins.
3.4 What you must not compare
eval_loss and mean_token_accuracy are not comparable between bases: ruadapt has a
different tokeniser, so both the number of tokens in an answer and the magnitude of the
loss are measured in different units. Within one base the curve is informative; across
bases it is not.
4. Repository layout
adapters/router-0.5b-vikhr-lora/ Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct + dataset v1
adapters/router-1.5b-ruadapt-lora/ RefalMachine/RuadaptQwen2.5-1.5B-instruct + dataset v1
adapters/router-1.7b-qvikhr3-lora/ Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning + dataset v1
MANIFEST.json sha256 of every file in this repo
Each is a PEFT adapter over its own base — the bases differ, so the subfolder and the
base model must be matched:
Each adapter directory carries its own train_summary.json (hyperparameters and training
metrics) and log_history.json (the loss curve). Intermediate checkpoint-*/ directories
with optimiser and RNG state were not uploaded.
There are no GGUF files here. None of these adapters was merged, converted or
quantised — there was no reason to build a deployment artefact for a candidate that lost.
Training hyperparameters are identical across all three and are listed in § 3; the only
variable is the base model.
5. Origin and licensing
Base models. Three different bases, each with its own licence, and this repository
does not restate them. Check each at its source before use:
The licences of these three were not verified as part of this work, and stating them from
memory would be exactly the kind of unsourced claim this project is built against. The
files here are LoRA deltas, not merged weights, so using them requires downloading the base
under its own terms anyway.
Training data. Generated programmatically from the JSON-schema catalogue of the
project's tools-core, which is GPL-3.0, inherited from it-tools
(tools-core/README.md). Tool ids, argument names, enum values and schema shapes in the
training corpus derive from that catalogue.
On the weights. Whether a copyleft licence on training data propagates to model weights
is an unsettled question in the industry, and this repository does not pretend to settle
it. We state the provenance and decline to declare the weights GPL-3.0. Not legal advice,
and not a grant.
6. Integrity
MANIFEST.json lists the sha256 of every published file, recorded at upload time on the
machine that produced them. The project tracks runs by weight hash rather than by tag: a
tag was once re-created from a different build while a 250-task run was in flight, and only
weights_sha256 made the swap visible (tracking/digit_tracking/artifacts.py).
7. What was not measured
GGUF quantisation, latency, memory and runtime behaviour — none of it, for any of
these three. They were never merged or converted.
Dataset v2 and v3 on these bases. The base comparison was deliberately run on v1 to
isolate base from data; nobody re-ran it afterwards, so it is unknown whether the ranking
survives the better dataset.
Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct and Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct
were never trained (§ 1.4).
Variance. One seed per base. 1–2 pp differences are inside the noise.
150 of the project's 400 eval tasks (rag_citation, fts_spec, multi_step) were
never run against any router — they need a corpus and the FTS compiler.
8. По-русски
Это проигравшие кандидаты. Репозиторий существует, чтобы отрицательный результат
остался воспроизводимым, а не превратился в фольклор. Поставочные модели —
в digit-router-0.6b и
digit-router-1.7b.
Это маршрутизаторы, а не ассистенты. Модель относит запрос к категории инструментов и
извлекает аргументы; содержание ответа даёт утилита, цитата или сертификат. Именно поэтому
русскоязычный instruct-тюнинг здесь и проигрывает: он улучшает свободную генерацию — ровно
то, чего архитектура не использует.
Главный вывод: русскоязычный SFT без перестроенного токенизатора не даёт ничего. Все
модели Vikhr дают ровно ноль выигрыша по fertility — не «небольшой», а ноль: цифры
совпадают с Qwen до четвёртого знака, потому что Vikhr не меняет словарь. То же верно для
QVikhr-3 поверх Qwen3. Ожидание «русская модель → русский токенизатор» здесь не выполняется,
и проверять это надо замером, а не по названию.
Выигрыш даёт только ruadapt, и он большой: 1,686 токена на слово против 2,344 на
русской прозе — −28,1 % — при словаре даже меньшего размера (145 152 против 151 669).
Все 14 схем шага 2 занимают 5 394 токена вместо 6 486, то есть в тот же контекст помещается
примерно на 20 % больше схем. Подтверждено дважды на реальном обучении: 14,85 млн токенов
за эпоху против 18,21 млн (−18,5 %).
Но ruadapt платит за это точностью аргументов. Лучшая маршрутизация (88,0 %), лучший
осознанный отказ (84,0 %) — и худшая среди крупных моделей точность аргументов: 85,7 %
против 92,7 % у Qwen3-1.7B. Это не артефакт разного числа попыток: на общем подмножестве
из 84 задач — 79,8 % против 85,7 %. Из 12 задач с неверными аргументами 10 — вторичные
параметры, и только 2 — основной литерал. В verified-архитектуре инструмент с неверным
аргументом выдаёт проверенный неверный ответ, поэтому точность аргументов весит больше,
и по главной метрике false_answer_rate выиграла Qwen/Qwen3-1.7B без всякой русификации.
Честная оговорка. Один seed на базу, датасет v1, 250 задач. Разница в 1–2 п.п. — это
2–5 задач, и она неотличима от случайности. Две модели из исходного списка
(Vikhr-Qwen-2.5-1.5B, Vikhr-Llama-3.2-1B) не обучались вовсе — аргумент против них
косвенный.
