| --- |
| license: other |
| license_name: see-per-adapter-base-model-licences |
| license_link: https://huggingface.co/digitable-lol/digit-router-experiments#5-origin-and-licensing |
| base_model: |
| - Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct |
| - RefalMachine/RuadaptQwen2.5-1.5B-instruct |
| - Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning |
| base_model_relation: adapter |
| language: |
| - ru |
| - en |
| tags: |
| - lora |
| - peft |
| - tool-use |
| - function-calling |
| - router |
| - russian |
| - ablation |
| --- |
| |
| # digit-router-experiments |
|
|
| **Three adapters that lost.** This repository exists so that a negative result stays |
| reproducible instead of becoming folklore. None of these is a shipping model; the models |
| that shipped are in |
| [`digitable-lol/digit-router-0.6b`](https://huggingface.co/digitable-lol/digit-router-0.6b) |
| and [`digitable-lol/digit-router-1.7b`](https://huggingface.co/digitable-lol/digit-router-1.7b). |
|
|
| --- |
|
|
| ## 1. Read this before you read any number |
|
|
| ### 1.1 These are routers, not assistants |
|
|
| Each adapter turns its base model into a two-step **tool router** for the Digit verified |
| agent: step 1 maps a query to one of 14 tool categories, step 2 emits a tool call with |
| extracted arguments given that category's JSON schemas — or refuses. The model never |
| produces the content of an answer; that comes from a deterministic utility, a verbatim |
| corpus quote, or a formal certificate. |
|
|
| This framing is the whole reason the Russian-SFT bases lose. Instruction tuning on Russian |
| dialogue improves free-form generation — the one capability this architecture deliberately |
| does not use. |
|
|
| ### 1.2 The headline result: Russian SFT without a rebuilt tokeniser buys nothing |
|
|
| **Every Vikhr model measured here gives exactly zero fertility gain over its own Qwen |
| base** — not "a small gain", zero: the numbers are identical to four decimal places, |
| because Vikhr is a Russian SFT over Qwen with an *unchanged* vocabulary. The same holds |
| for QVikhr-3 over Qwen3. The expectation "Russian model → Russian tokeniser" simply does |
| not hold, and it has to be checked by measurement rather than by the name. |
|
|
| The only base that gains is `ruadapt`, which actually extends the vocabulary — and it pays |
| for the gain in argument accuracy. Details in § 2 and § 3. |
|
|
| ### 1.3 A counted refusal is not a conscious refusal |
|
|
| The eval harness scores an empty or unparseable answer as a refusal, so a model that |
| simply breaks scores like a model that knows when to decline. The refusal column below is |
| the **conscious** one: the router emitted an explicit `{"refuse": "..."}` object. The |
| `unreadable` column is what the counted number would have absorbed. For reference, the |
| untuned `Qwen3-0.6B` base scores 75.3 % counted against **9.3 %** conscious, with 115 |
| unreadable answers out of 250 — a 66 pp gap that is entirely broken output. |
|
|
| ### 1.4 Single seed, dataset v1, and one candidate pair is missing |
|
|
| Every adapter here was trained **once**, with `seed=20260802`, on **dataset v1** |
| (deliberately: v1 isolates the effect of the base from the effect of the data). Differences |
| of 1–2 pp on 250 tasks are 2–5 tasks and are not distinguishable from initialisation noise. |
|
|
| Two candidates in the original shortlist were **never trained**: |
| `Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct` (same tokeniser and same nature as the Vikhr |
| models that were tested) and `Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct` (three times worse |
| fertility than ruadapt). The argument against both is indirect, and it is stated here |
| rather than hidden: strictly speaking they are two untested candidates. |
|
|
| --- |
|
|
| ## 2. Measurement 1 — tokeniser fertility |
|
|
| The only number about a base model obtainable without training anything. It answers two |
| practical questions at once: how much work a request costs (a token is the unit of work) |
| and how many tool schemas fit in the context budget. |
|
|
| Corpus: three slices of real data, not "typical Russian text" — 3 000 paragraphs of Russian |
| course prose (941 376 characters, 136 722 words), 4 000 real user queries (186 289 |
| characters), and the 14 step-2 system prompts (19 640 characters) that determine how many |
| tools fit in context. |
|
|
| | Model | Vocab | RU prose, tok/word | chars/tok | Queries, tok/word | Step-2 schemas, total tokens | median schema | |
| |---|---:|---:|---:|---:|---:|---:| |
| | `Qwen/Qwen3-0.6B` (control) | 151 669 | 2.344 | 2.94 | 3.056 | 6 486 | 487 | |
| | `Qwen/Qwen3-1.7B` | 151 669 | 2.344 | 2.94 | 3.056 | 6 486 | 487 | |
| | `Qwen/Qwen2.5-0.5B-Instruct` | 151 665 | 2.344 | 2.94 | 3.056 | 6 486 | 487 | |
| | `Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct` | 151 665 | 2.344 | 2.94 | 3.056 | 6 486 | 487 | |
| | `Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct` | 151 665 | 2.344 | 2.94 | 3.056 | 6 486 | 487 | |
| | `Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning` | 151 669 | 2.344 | 2.94 | 3.056 | 6 486 | 487 | |
| | **`RefalMachine/RuadaptQwen2.5-1.5B-instruct`** | 145 152 | **1.686 (−28.1 %)** | **4.08** | **2.597 (−15.0 %)** | **5 394 (−16.8 %)** | **407** | |
| | `Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct` | 128 256 | 2.177 (−7.1 %) | 3.16 | 2.808 (−8.1 %) | 6 258 (−3.5 %) | 466 | |
|
|
| Look at the six identical rows. Four different "Russian" models and two Qwen controls |
| produce byte-identical fertility, because none of them touched the vocabulary. |
|
|
| **Only `ruadapt` gains, and it gains a lot**: 1.686 tokens per word against 2.344 on |
| Russian prose — 28 % fewer — with a *smaller* vocabulary (145 152 against 151 669). On our |
| queries the gap is narrower (−15 %) because queries are short and full of Latin script, |
| numbers and identifiers, where there is nothing to gain. In context terms, all 14 step-2 |
| schemas take 5 394 tokens instead of 6 486, so roughly 20 % more schemas fit the same |
| budget. |
|
|
| The figure is confirmed twice on real training: ruadapt's median training example is 447 |
| tokens against 562 for Qwen3 (p99 709 against 811), and one epoch over the same dataset |
| passes **14.85 M tokens against 18.21 M** — 18.5 % fewer — in 56.4 minutes against 71.8. |
|
|
| *Source: `train/RESULTS.md` § 2, raw numbers in `train/fertility/fertility.json`.* |
|
|
| --- |
|
|
| ## 3. Measurement 2 — quality, identical LoRA on five bases |
|
|
| Everything identical except the base model: same dataset (**v1**), same LoRA (`r=32`, |
| `alpha=64`, dropout 0.05, the same seven target modules), `lr=1e-4` cosine with 3 % warmup, |
| effective batch 16 (4 × accumulation 4), `max_length=1024`, `adamw_bnb_8bit`, one epoch, |
| `seed=20260802`, greedy decoding, the same two-step inference. All 250 tasks, no sampling, |
| unmodified `eval/scoring.py`. |
|
|
| | Base | Params | tool_accuracy | arg_accuracy | **conscious refusal (red-team)** | false_answer_rate | over_refusal | unreadable | training | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:| |
| | `Qwen3-0.6B` (incumbent) | 0.6B | 83.0 % | **93.7 %** | 75.3 % | 17.2 % | 12.0 % | 7 | 35.7 min | |
| | **`Vikhr-Qwen-2.5-0.5b`** ← in this repo | 0.5B | 81.0 % | 83.8 % | 63.3 % | 27.2 % | 11.0 % | 11 | 24.7 min | |
| | **`RuadaptQwen2.5-1.5B`** ← in this repo | 1.5B | **88.0 %** | 85.7 % | **84.0 %** | 14.8 % | **9.0 %** | 4 | 56.4 min | |
| | **`QVikhr-3-1.7B`** ← in this repo | 1.7B | 84.0 % | 90.1 % | 82.7 % | 15.6 % | 11.0 % | **3** | 72.2 min | |
| | **`Qwen3-1.7B`** (winner) | 1.7B | 86.0 % | 92.7 % | 83.3 % | **12.8 %** | **9.0 %** | 4 | 71.8 min | |
| |
| On the two gaps the whole exercise was about: |
| |
| | Base | `false_premise`: answered with a tool call | `missing_param`: answered with a tool call | |
| |---|---:|---:| |
| | `Qwen3-0.6B` | 13/30 | 16/40 | |
| | `Vikhr-Qwen-2.5-0.5b` | 13/30 | 22/40 | |
| | `RuadaptQwen2.5-1.5B` | 9/30 | **12/40** | |
| | `QVikhr-3-1.7B` | 11/30 | 13/40 | |
| | **`Qwen3-1.7B`** | **6/30** | 15/40 | |
| |
| ### 3.1 Russian SFT does not help — it hurts, and it shows on two honest pairs |
| |
| * **`Vikhr-Qwen-2.5-0.5b` vs `Qwen3-0.6B`** (roughly one size): worse on everything — |
| routing 81 vs 83, arguments **83.8 vs 93.7**, conscious refusal **63.3 vs 75.3**, |
| `false_answer_rate` **27.2 vs 17.2**, decoys 22/40 vs 16/40. On the main diagnosed gap |
| it is markedly worse. This is the direct answer to the original proposal "take Vikhr on |
| a Qwen2.5-0.5B base": the measured answer is don't. |
| * **`QVikhr-3-1.7B` vs `Qwen3-1.7B`** — the strong pair, because this is **the same base |
| model**, the same tokeniser and the same size, differing only by a Russian SFT layer on |
| top. Worse on everything that matters: `false_answer_rate` 15.6 vs 12.8, arguments |
| 90.1 vs 92.7, false premises 11/30 vs 6/30. The Russian SFT added nothing and blurred |
| part of the original capability. |
| |
| The cause follows from § 1.1: in a verified architecture the model classifies and extracts, |
| it does not write. Instruction tuning on Russian dialogue improves what we do not use. |
| |
| ### 3.2 An extended Russian tokeniser is the only thing that really works — and it is not free |
| |
| `ruadapt` takes the best routing (88.0 %), the best conscious refusal (84.0 %) and the best |
| decoy result (12/40) — and the worst argument accuracy among the large models: |
| **85.7 % against 92.7 %**. |
| |
| This is not an artefact of ruadapt attempting more tasks. On the **84 tasks both models |
| attempted**, `Qwen3-0.6B` gets 85.7 % of arguments right and `ruadapt` 79.8 %. Of ruadapt's |
| 12 wrong-argument tasks, **10 are secondary parameters** (`hashFunction`, `cypherAlgo`, |
| `font`, `paragraphs`, `indentSize`, `timeSpan`, `defaultCountryCode`) and only 2 are the |
| main literal (`inputMarkdown`, `rawPhone`). The model confidently identifies *which* tool |
| is needed and fills its settings worse. |
| |
| ### 3.3 Why argument accuracy outweighed routing accuracy |
| |
| Choosing between `Qwen3-1.7B` and `ruadapt` was choosing between 6 argument tasks and |
| 2 routing tasks. In a verified architecture a tool called with a wrong argument returns a |
| **verified wrong answer**; a wrongly chosen tool usually fails its own validation and is |
| visible. So argument accuracy weighs more — and it agrees with the harness's primary |
| metric, `false_answer_rate`, on which `Qwen3-1.7B` wins. |
| |
| ### 3.4 What you must not compare |
| |
| `eval_loss` and `mean_token_accuracy` are **not comparable between bases**: ruadapt has a |
| different tokeniser, so both the number of tokens in an answer and the magnitude of the |
| loss are measured in different units. Within one base the curve is informative; across |
| bases it is not. |
|
|
| --- |
|
|
| ## 4. Repository layout |
|
|
| ``` |
| adapters/router-0.5b-vikhr-lora/ Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct + dataset v1 |
| adapters/router-1.5b-ruadapt-lora/ RefalMachine/RuadaptQwen2.5-1.5B-instruct + dataset v1 |
| adapters/router-1.7b-qvikhr3-lora/ Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning + dataset v1 |
| MANIFEST.json sha256 of every file in this repo |
| ``` |
|
|
| Each is a PEFT adapter over its own base — the bases differ, so the `subfolder` and the |
| base model must be matched: |
|
|
| ```python |
| from peft import PeftModel |
| from transformers import AutoModelForCausalLM |
| |
| base = AutoModelForCausalLM.from_pretrained("RefalMachine/RuadaptQwen2.5-1.5B-instruct", |
| torch_dtype="bfloat16") |
| model = PeftModel.from_pretrained(base, "digitable-lol/digit-router-experiments", |
| subfolder="adapters/router-1.5b-ruadapt-lora") |
| ``` |
|
|
| Each adapter directory carries its own `train_summary.json` (hyperparameters and training |
| metrics) and `log_history.json` (the loss curve). Intermediate `checkpoint-*/` directories |
| with optimiser and RNG state were **not** uploaded. |
|
|
| There are **no GGUF files here.** None of these adapters was merged, converted or |
| quantised — there was no reason to build a deployment artefact for a candidate that lost. |
|
|
| Training hyperparameters are identical across all three and are listed in § 3; the only |
| variable is the base model. |
|
|
| --- |
|
|
| ## 5. Origin and licensing |
|
|
| **Base models.** Three different bases, each with its own licence, and **this repository |
| does not restate them.** Check each at its source before use: |
|
|
| * [`Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct`](https://huggingface.co/Vikhrmodels/Vikhr-Qwen-2.5-0.5b-Instruct) |
| * [`RefalMachine/RuadaptQwen2.5-1.5B-instruct`](https://huggingface.co/RefalMachine/RuadaptQwen2.5-1.5B-instruct) |
| * [`Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning`](https://huggingface.co/Vikhrmodels/QVikhr-3-1.7B-Instruction-noreasoning) |
|
|
| The licences of these three were not verified as part of this work, and stating them from |
| memory would be exactly the kind of unsourced claim this project is built against. The |
| files here are LoRA deltas, not merged weights, so using them requires downloading the base |
| under its own terms anyway. |
|
|
| **Training data.** Generated programmatically from the JSON-schema catalogue of the |
| project's `tools-core`, which is **GPL-3.0, inherited from `it-tools`** |
| (`tools-core/README.md`). Tool ids, argument names, enum values and schema shapes in the |
| training corpus derive from that catalogue. |
|
|
| **On the weights.** Whether a copyleft licence on training data propagates to model weights |
| is an unsettled question in the industry, and this repository does not pretend to settle |
| it. We state the provenance and decline to declare the weights GPL-3.0. Not legal advice, |
| and not a grant. |
|
|
| --- |
|
|
| ## 6. Integrity |
|
|
| `MANIFEST.json` lists the sha256 of every published file, recorded at upload time on the |
| machine that produced them. The project tracks runs by weight hash rather than by tag: a |
| tag was once re-created from a different build while a 250-task run was in flight, and only |
| `weights_sha256` made the swap visible (`tracking/digit_tracking/artifacts.py`). |
|
|
| --- |
|
|
| ## 7. What was not measured |
|
|
| * **GGUF quantisation, latency, memory and runtime behaviour** — none of it, for any of |
| these three. They were never merged or converted. |
| * **Dataset v2 and v3 on these bases.** The base comparison was deliberately run on v1 to |
| isolate base from data; nobody re-ran it afterwards, so it is unknown whether the ranking |
| survives the better dataset. |
| * **`Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct` and `Vikhrmodels/Vikhr-Llama-3.2-1B-Instruct` |
| were never trained** (§ 1.4). |
| * **Variance.** One seed per base. 1–2 pp differences are inside the noise. |
| * **150 of the project's 400 eval tasks** (`rag_citation`, `fts_spec`, `multi_step`) were |
| never run against any router — they need a corpus and the FTS compiler. |
|
|
| --- |
|
|
| ## 8. По-русски |
|
|
| **Это проигравшие кандидаты.** Репозиторий существует, чтобы отрицательный результат |
| остался воспроизводимым, а не превратился в фольклор. Поставочные модели — |
| в [`digit-router-0.6b`](https://huggingface.co/digitable-lol/digit-router-0.6b) и |
| [`digit-router-1.7b`](https://huggingface.co/digitable-lol/digit-router-1.7b). |
|
|
| **Это маршрутизаторы, а не ассистенты.** Модель относит запрос к категории инструментов и |
| извлекает аргументы; содержание ответа даёт утилита, цитата или сертификат. Именно поэтому |
| русскоязычный instruct-тюнинг здесь и проигрывает: он улучшает свободную генерацию — ровно |
| то, чего архитектура не использует. |
|
|
| **Главный вывод: русскоязычный SFT без перестроенного токенизатора не даёт ничего.** Все |
| модели Vikhr дают ровно ноль выигрыша по fertility — не «небольшой», а ноль: цифры |
| совпадают с Qwen до четвёртого знака, потому что Vikhr не меняет словарь. То же верно для |
| QVikhr-3 поверх Qwen3. Ожидание «русская модель → русский токенизатор» здесь не выполняется, |
| и проверять это надо замером, а не по названию. |
|
|
| **Выигрыш даёт только `ruadapt`, и он большой:** 1,686 токена на слово против 2,344 на |
| русской прозе — **−28,1 %** — при словаре даже меньшего размера (145 152 против 151 669). |
| Все 14 схем шага 2 занимают 5 394 токена вместо 6 486, то есть в тот же контекст помещается |
| примерно на 20 % больше схем. Подтверждено дважды на реальном обучении: 14,85 млн токенов |
| за эпоху против 18,21 млн (−18,5 %). |
|
|
| **Но `ruadapt` платит за это точностью аргументов.** Лучшая маршрутизация (88,0 %), лучший |
| осознанный отказ (84,0 %) — и худшая среди крупных моделей точность аргументов: **85,7 % |
| против 92,7 %** у Qwen3-1.7B. Это не артефакт разного числа попыток: на общем подмножестве |
| из 84 задач — 79,8 % против 85,7 %. Из 12 задач с неверными аргументами 10 — вторичные |
| параметры, и только 2 — основной литерал. В verified-архитектуре инструмент с неверным |
| аргументом выдаёт **проверенный неверный ответ**, поэтому точность аргументов весит больше, |
| и по главной метрике `false_answer_rate` выиграла `Qwen/Qwen3-1.7B` без всякой русификации. |
|
|
| **Честная оговорка.** Один seed на базу, датасет v1, 250 задач. Разница в 1–2 п.п. — это |
| 2–5 задач, и она неотличима от случайности. Две модели из исходного списка |
| (`Vikhr-Qwen-2.5-1.5B`, `Vikhr-Llama-3.2-1B`) не обучались вовсе — аргумент против них |
| косвенный. |
|
|
