Hugging Face's logo

digitable-lol
/
digit-router-experiments

PEFT
Safetensors
Russian
English
lora
tool-use
function-calling
router
russian
ablation
Model card Files
xet
 Community
digit-router-experiments / README.md
the-homeless-god's picture
the-homeless-god
publish digit-router-experiments: 29 files (374.8 MiB)
27073bb verified
preview code
|
Raw
History Blame Contribute Delete
18.1 kB
metadata 
license: other
license_name: see-per-adapter-base-model-licences
license_link: >-
  https://huggingface.co/digitable-lol/digit-router-experiments#5-origin-and-licensing
base_model:
  - Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct
  - RefalMachine/RuadaptQwen2.5-1.5B-instruct
  - Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning
base_model_relation: adapter
language:
  - ru
  - en
tags:
  - lora
  - peft
  - tool-use
  - function-calling
  - router
  - russian
  - ablation

digit-router-experiments

Three adapters that lost. This repository exists so that a negative result stays reproducible instead of becoming folklore. None of these is a shipping model; the models that shipped are in digitable-lol/digit-router-0.6b and digitable-lol/digit-router-1.7b.

1. Read this before you read any number

1.1 These are routers, not assistants

Each adapter turns its base model into a two-step tool router for the Digit verified agent: step 1 maps a query to one of 14 tool categories, step 2 emits a tool call with extracted arguments given that category's JSON schemas — or refuses. The model never produces the content of an answer; that comes from a deterministic utility, a verbatim corpus quote, or a formal certificate.

This framing is the whole reason the Russian-SFT bases lose. Instruction tuning on Russian dialogue improves free-form generation — the one capability this architecture deliberately does not use.

1.2 The headline result: Russian SFT without a rebuilt tokeniser buys nothing

Every Vikhr model measured here gives exactly zero fertility gain over its own Qwen base — not "a small gain", zero: the numbers are identical to four decimal places, because Vikhr is a Russian SFT over Qwen with an unchanged vocabulary. The same holds for QVikhr-3 over Qwen3. The expectation "Russian model → Russian tokeniser" simply does not hold, and it has to be checked by measurement rather than by the name.

The only base that gains is ruadapt, which actually extends the vocabulary — and it pays for the gain in argument accuracy. Details in § 2 and § 3.

1.3 A counted refusal is not a conscious refusal

The eval harness scores an empty or unparseable answer as a refusal, so a model that simply breaks scores like a model that knows when to decline. The refusal column below is the conscious one: the router emitted an explicit {"refuse": "..."} object. The unreadable column is what the counted number would have absorbed. For reference, the untuned Qwen3-0.6B base scores 75.3 % counted against 9.3 % conscious, with 115 unreadable answers out of 250 — a 66 pp gap that is entirely broken output.

1.4 Single seed, dataset v1, and one candidate pair is missing

Every adapter here was trained once, with seed=20260802, on dataset v1 (deliberately: v1 isolates the effect of the base from the effect of the data). Differences of 1–2 pp on 250 tasks are 2–5 tasks and are not distinguishable from initialisation noise.

Two candidates in the original shortlist were never trained: Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct (same tokeniser and same nature as the Vikhr models that were tested) and Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct (three times worse fertility than ruadapt). The argument against both is indirect, and it is stated here rather than hidden: strictly speaking they are two untested candidates.

2. Measurement 1 — tokeniser fertility

The only number about a base model obtainable without training anything. It answers two practical questions at once: how much work a request costs (a token is the unit of work) and how many tool schemas fit in the context budget.

Corpus: three slices of real data, not "typical Russian text" — 3 000 paragraphs of Russian course prose (941 376 characters, 136 722 words), 4 000 real user queries (186 289 characters), and the 14 step-2 system prompts (19 640 characters) that determine how many tools fit in context.

Model Vocab RU prose, tok/word chars/tok Queries, tok/word Step-2 schemas, total tokens median schema
Qwen/Qwen3-0.6B (control) 151 669 2.344 2.94 3.056 6 486 487
Qwen/Qwen3-1.7B 151 669 2.344 2.94 3.056 6 486 487
Qwen/Qwen2.5-0.5B-Instruct 151 665 2.344 2.94 3.056 6 486 487
Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct 151 665 2.344 2.94 3.056 6 486 487
Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct 151 665 2.344 2.94 3.056 6 486 487
Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning 151 669 2.344 2.94 3.056 6 486 487
RefalMachine/RuadaptQwen2.5-1.5B-instruct 145 152 1.686 (−28.1 %) 4.08 2.597 (−15.0 %) 5 394 (−16.8 %) 407
Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct 128 256 2.177 (−7.1 %) 3.16 2.808 (−8.1 %) 6 258 (−3.5 %) 466

Look at the six identical rows. Four different "Russian" models and two Qwen controls produce byte-identical fertility, because none of them touched the vocabulary.

Only ruadapt gains, and it gains a lot: 1.686 tokens per word against 2.344 on Russian prose — 28 % fewer — with a smaller vocabulary (145 152 against 151 669). On our queries the gap is narrower (−15 %) because queries are short and full of Latin script, numbers and identifiers, where there is nothing to gain. In context terms, all 14 step-2 schemas take 5 394 tokens instead of 6 486, so roughly 20 % more schemas fit the same budget.

The figure is confirmed twice on real training: ruadapt's median training example is 447 tokens against 562 for Qwen3 (p99 709 against 811), and one epoch over the same dataset passes 14.85 M tokens against 18.21 M — 18.5 % fewer — in 56.4 minutes against 71.8.

Source: train/RESULTS.md § 2, raw numbers in train/fertility/fertility.json.

3. Measurement 2 — quality, identical LoRA on five bases

Everything identical except the base model: same dataset (v1), same LoRA (r=32, alpha=64, dropout 0.05, the same seven target modules), lr=1e-4 cosine with 3 % warmup, effective batch 16 (4 × accumulation 4), max_length=1024, adamw_bnb_8bit, one epoch, seed=20260802, greedy decoding, the same two-step inference. All 250 tasks, no sampling, unmodified eval/scoring.py.

Base Params tool_accuracy arg_accuracy conscious refusal (red-team) false_answer_rate over_refusal unreadable training
Qwen3-0.6B (incumbent) 0.6B 83.0 % 93.7 % 75.3 % 17.2 % 12.0 % 7 35.7 min
Vikhr-Qwen-2.5-0.5b ← in this repo 0.5B 81.0 % 83.8 % 63.3 % 27.2 % 11.0 % 11 24.7 min
RuadaptQwen2.5-1.5B ← in this repo 1.5B 88.0 % 85.7 % 84.0 % 14.8 % 9.0 % 4 56.4 min
QVikhr-3-1.7B ← in this repo 1.7B 84.0 % 90.1 % 82.7 % 15.6 % 11.0 % 3 72.2 min
Qwen3-1.7B (winner) 1.7B 86.0 % 92.7 % 83.3 % 12.8 % 9.0 % 4 71.8 min

On the two gaps the whole exercise was about:

Base false_premise: answered with a tool call missing_param: answered with a tool call
Qwen3-0.6B 13/30 16/40
Vikhr-Qwen-2.5-0.5b 13/30 22/40
RuadaptQwen2.5-1.5B 9/30 12/40
QVikhr-3-1.7B 11/30 13/40
Qwen3-1.7B 6/30 15/40

3.1 Russian SFT does not help — it hurts, and it shows on two honest pairs

  • Vikhr-Qwen-2.5-0.5b vs Qwen3-0.6B (roughly one size): worse on everything — routing 81 vs 83, arguments 83.8 vs 93.7, conscious refusal 63.3 vs 75.3, false_answer_rate 27.2 vs 17.2, decoys 22/40 vs 16/40. On the main diagnosed gap it is markedly worse. This is the direct answer to the original proposal "take Vikhr on a Qwen2.5-0.5B base": the measured answer is don't.
  • QVikhr-3-1.7B vs Qwen3-1.7B — the strong pair, because this is the same base model, the same tokeniser and the same size, differing only by a Russian SFT layer on top. Worse on everything that matters: false_answer_rate 15.6 vs 12.8, arguments 90.1 vs 92.7, false premises 11/30 vs 6/30. The Russian SFT added nothing and blurred part of the original capability.

The cause follows from § 1.1: in a verified architecture the model classifies and extracts, it does not write. Instruction tuning on Russian dialogue improves what we do not use.

3.2 An extended Russian tokeniser is the only thing that really works — and it is not free

ruadapt takes the best routing (88.0 %), the best conscious refusal (84.0 %) and the best decoy result (12/40) — and the worst argument accuracy among the large models: 85.7 % against 92.7 %.

This is not an artefact of ruadapt attempting more tasks. On the 84 tasks both models attempted, Qwen3-0.6B gets 85.7 % of arguments right and ruadapt 79.8 %. Of ruadapt's 12 wrong-argument tasks, 10 are secondary parameters (hashFunction, cypherAlgo, font, paragraphs, indentSize, timeSpan, defaultCountryCode) and only 2 are the main literal (inputMarkdown, rawPhone). The model confidently identifies which tool is needed and fills its settings worse.

3.3 Why argument accuracy outweighed routing accuracy

Choosing between Qwen3-1.7B and ruadapt was choosing between 6 argument tasks and 2 routing tasks. In a verified architecture a tool called with a wrong argument returns a verified wrong answer; a wrongly chosen tool usually fails its own validation and is visible. So argument accuracy weighs more — and it agrees with the harness's primary metric, false_answer_rate, on which Qwen3-1.7B wins.

3.4 What you must not compare

eval_loss and mean_token_accuracy are not comparable between bases: ruadapt has a different tokeniser, so both the number of tokens in an answer and the magnitude of the loss are measured in different units. Within one base the curve is informative; across bases it is not.

4. Repository layout 

adapters/router-0.5b-vikhr-lora/      Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct     + dataset v1
adapters/router-1.5b-ruadapt-lora/    RefalMachine/RuadaptQwen2.5-1.5B-instruct    + dataset v1
adapters/router-1.7b-qvikhr3-lora/    Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning + dataset v1
MANIFEST.json                         sha256 of every file in this repo

Each is a PEFT adapter over its own base — the bases differ, so the subfolder and the base model must be matched:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("RefalMachine/RuadaptQwen2.5-1.5B-instruct",
                                            torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "digitable-lol/digit-router-experiments",
                                  subfolder="adapters/router-1.5b-ruadapt-lora")

Each adapter directory carries its own train_summary.json (hyperparameters and training metrics) and log_history.json (the loss curve). Intermediate checkpoint-*/ directories with optimiser and RNG state were not uploaded.

There are no GGUF files here. None of these adapters was merged, converted or quantised — there was no reason to build a deployment artefact for a candidate that lost.

Training hyperparameters are identical across all three and are listed in § 3; the only variable is the base model.

5. Origin and licensing

Base models. Three different bases, each with its own licence, and this repository does not restate them. Check each at its source before use:

The licences of these three were not verified as part of this work, and stating them from memory would be exactly the kind of unsourced claim this project is built against. The files here are LoRA deltas, not merged weights, so using them requires downloading the base under its own terms anyway.

Training data. Generated programmatically from the JSON-schema catalogue of the project's tools-core, which is GPL-3.0, inherited from it-tools (tools-core/README.md). Tool ids, argument names, enum values and schema shapes in the training corpus derive from that catalogue.

On the weights. Whether a copyleft licence on training data propagates to model weights is an unsettled question in the industry, and this repository does not pretend to settle it. We state the provenance and decline to declare the weights GPL-3.0. Not legal advice, and not a grant.

6. Integrity

MANIFEST.json lists the sha256 of every published file, recorded at upload time on the machine that produced them. The project tracks runs by weight hash rather than by tag: a tag was once re-created from a different build while a 250-task run was in flight, and only weights_sha256 made the swap visible (tracking/digit_tracking/artifacts.py).

7. What was not measured

  • GGUF quantisation, latency, memory and runtime behaviour — none of it, for any of these three. They were never merged or converted.
  • Dataset v2 and v3 on these bases. The base comparison was deliberately run on v1 to isolate base from data; nobody re-ran it afterwards, so it is unknown whether the ranking survives the better dataset.
  • Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct and Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct were never trained (§ 1.4).
  • Variance. One seed per base. 1–2 pp differences are inside the noise.
  • 150 of the project's 400 eval tasks (rag_citation, fts_spec, multi_step) were never run against any router — they need a corpus and the FTS compiler.

8. По-русски

Это проигравшие кандидаты. Репозиторий существует, чтобы отрицательный результат остался воспроизводимым, а не превратился в фольклор. Поставочные модели — в digit-router-0.6b и digit-router-1.7b.

Это маршрутизаторы, а не ассистенты. Модель относит запрос к категории инструментов и извлекает аргументы; содержание ответа даёт утилита, цитата или сертификат. Именно поэтому русскоязычный instruct-тюнинг здесь и проигрывает: он улучшает свободную генерацию — ровно то, чего архитектура не использует.

Главный вывод: русскоязычный SFT без перестроенного токенизатора не даёт ничего. Все модели Vikhr дают ровно ноль выигрыша по fertility — не «небольшой», а ноль: цифры совпадают с Qwen до четвёртого знака, потому что Vikhr не меняет словарь. То же верно для QVikhr-3 поверх Qwen3. Ожидание «русская модель → русский токенизатор» здесь не выполняется, и проверять это надо замером, а не по названию.

Выигрыш даёт только ruadapt, и он большой: 1,686 токена на слово против 2,344 на русской прозе — −28,1 % — при словаре даже меньшего размера (145 152 против 151 669). Все 14 схем шага 2 занимают 5 394 токена вместо 6 486, то есть в тот же контекст помещается примерно на 20 % больше схем. Подтверждено дважды на реальном обучении: 14,85 млн токенов за эпоху против 18,21 млн (−18,5 %).

Но ruadapt платит за это точностью аргументов. Лучшая маршрутизация (88,0 %), лучший осознанный отказ (84,0 %) — и худшая среди крупных моделей точность аргументов: 85,7 % против 92,7 % у Qwen3-1.7B. Это не артефакт разного числа попыток: на общем подмножестве из 84 задач — 79,8 % против 85,7 %. Из 12 задач с неверными аргументами 10 — вторичные параметры, и только 2 — основной литерал. В verified-архитектуре инструмент с неверным аргументом выдаёт проверенный неверный ответ, поэтому точность аргументов весит больше, и по главной метрике false_answer_rate выиграла Qwen/Qwen3-1.7B без всякой русификации.

Честная оговорка. Один seed на базу, датасет v1, 250 задач. Разница в 1–2 п.п. — это 2–5 задач, и она неотличима от случайности. Две модели из исходного списка (Vikhr-Qwen-2.5-1.5B, Vikhr-Llama-3.2-1B) не обучались вовсе — аргумент против них косвенный.