the-homeless-god commited on
Commit
75d23d7
·
verified ·
1 Parent(s): 3dd4fda

publish README: 2 files (10.7 KiB)

Browse files

sha256 of every file is in MANIFEST.json

Files changed (2) hide show
  1. MANIFEST.json +13 -0
  2. README.md +82 -29
MANIFEST.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "generated_at": "2026-08-07T09:09:59+00:00",
3
+ "repo_id": "digitable-lol/README",
4
+ "repo_type": "model",
5
+ "note": "sha256 of every file as published. The hash is the record, not the filename: a repository is the same artefact as this manifest only if every digest matches.",
6
+ "files": [
7
+ {
8
+ "path_in_repo": "README.md",
9
+ "sha256": "9d960acba49b6387c926ad6ce034b60d52efe5d38defe3c89fd170212daf1975",
10
+ "size_bytes": 10491
11
+ }
12
+ ]
13
+ }
README.md CHANGED
@@ -4,46 +4,81 @@ We build **Digit**, a local agent for the Russian-language course platform
4
  [courses.digitable.life](https://courses.digitable.life). Its design constraint is
5
  unusual: the language model is not allowed to be a source of facts. Every fact in an
6
  answer comes from a deterministic utility, a verbatim quote from the course corpus, or a
7
- formal certificate — the model only chooses which of those to invoke.
8
-
9
- The models published here are the piece that does the choosing.
10
 
11
  ## What is in this organisation
12
 
13
- **These are tool routers, not general-purpose assistants.** Each one maps a user query to
14
- a tool category, then emits a tool call with extracted arguments — or refuses. It does not
15
- write the answer. Load one as a chat model and you will get nonsense, and none of our
16
- published metrics describe that use.
 
 
 
 
 
 
 
 
17
 
18
  | Repository | What it is |
19
  |---|---|
20
  | [`digit-router-0.6b`](https://huggingface.co/digitable-lol/digit-router-0.6b) | The shipping router. LoRA adapters (v1/v2/v3) over `Qwen/Qwen3-0.6B` plus merged GGUF quantisations. 424 MiB at Q5_K_M, ~500 ms per full two-step routing cycle on 8 CPU threads. |
21
  | [`digit-router-1.7b`](https://huggingface.co/digitable-lol/digit-router-1.7b) | The same training run at 1.7B. Higher routing accuracy; measurably not worth 3× the parameters for this task. |
22
  | [`digit-router-experiments`](https://huggingface.co/digitable-lol/digit-router-experiments) | Three adapters that lost — Vikhr, ruadapt, QVikhr-3. Published so the negative result stays reproducible instead of becoming folklore. |
 
 
 
 
 
 
 
 
 
 
23
 
24
  ## How to read our numbers
25
 
26
- Two conventions run through every model card, and both exist because the obvious way to
27
  report these numbers is misleading:
28
 
29
  * **A counted refusal is not a conscious refusal.** An eval harness scores an unparseable
30
  answer as a refusal, so a model that merely breaks scores like a model that knows when
31
  to decline. We always report both columns. The untuned 0.6B base scores 75.3 % counted
32
  against 9.3 % conscious — a 66-point gap that is entirely broken output.
 
 
 
 
 
 
 
 
 
33
  * **Known defects are stated before the good tables, not in a footnote.** The imatrix
34
- quantisation of the 0.6B model measurably breaks its ability to refuse and carries a
35
- do-not-deploy warning next to the file. At Q4 the routers do not emit garbage; they emit
36
- structurally flawless tool calls with invented arguments, and a GBNF grammar does not
37
- catch that. Our v3 adapters are a routing regression against v2 on a single seed. All of
38
- this is on the model pages.
 
 
 
 
 
 
 
39
 
40
  Every published file's sha256 is recorded in a `MANIFEST.json` in its repository. We track
41
  runs by weight hash rather than by tag, because a tag was once re-created from a different
42
  build while a 250-task evaluation was in flight.
43
 
44
- Base models are `Qwen/Qwen3-*` under Apache-2.0. The training data is derived from a
45
  GPL-3.0 utility catalogue; we state that provenance on every page and do not claim to have
46
- resolved what it means for weights.
 
 
47
 
48
  ---
49
 
@@ -53,20 +88,38 @@ resolved what it means for weights.
53
  [courses.digitable.life](https://courses.digitable.life). Ограничение архитектуры
54
  необычное: языковой модели запрещено быть источником фактов. Содержание ответа даёт
55
  детерминированная утилита, дословная цитата из корпуса курсов или формальный сертификат.
56
- Модель выбирает, что вызвать, — и только.
57
-
58
- **Здесь лежат маршрутизаторы, а не универсальные ассистенты.** Модель относит запрос к
59
- категории инструментов и извлекает аргументы либо отказывается; ответ она не пишет. Если
60
- загрузить её как чат-модель, вы получите бессмыслицу, и опубликованные метрики к такому
61
- использованию не относятся.
62
-
63
- Два правила чтения наших чисел. Первое: **засчитанный отказ ≠ осознанный** — харнесс
64
- считает отказом любой неразбираемый ответ, поэтому сломанная модель выглядит как
65
- осторожная; мы всегда печатаем обе колонки. Второе: **известные дефекты стоят до таблиц с
66
- хорошими числами, а не в примечаниях.** imatrix-квант 0.6B ломает способность отказываться
67
- и помечен как непригодный к поставке; при Q4 модель выдаёт структурно безупречные вызовы с
68
- выдуманными аргументами, и грамматика этого не ловит; адаптеры v3 — измеренный регресс
69
- маршрутизации против v2 на одном seed.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70
 
71
  sha256 каждого опубликованного файла записан в `MANIFEST.json` соответствующего
72
  репозитория.
 
4
  [courses.digitable.life](https://courses.digitable.life). Its design constraint is
5
  unusual: the language model is not allowed to be a source of facts. Every fact in an
6
  answer comes from a deterministic utility, a verbatim quote from the course corpus, or a
7
+ formal certificate — the model only chooses which of those to invoke, or writes the
8
+ artefact that a checker then verifies.
 
9
 
10
  ## What is in this organisation
11
 
12
+ Two lines of work, and neither produces a general-purpose assistant.
13
+
14
+ **Routers** map a user query to a tool category and emit a tool call with extracted
15
+ arguments — or refuse. They never write the content of an answer. Load one as a chat
16
+ model and you will get nonsense, and none of our published metrics describe that use.
17
+
18
+ **The specification generator** writes executable
19
+ [FTS](https://github.com/digitable-lol/flang) specifications from a task stated in
20
+ ordinary Russian. Its outputs are not trusted either: they are compiled, type checked,
21
+ executed and proved by a real toolchain, and only what survives counts.
22
+
23
+ ### Models
24
 
25
  | Repository | What it is |
26
  |---|---|
27
  | [`digit-router-0.6b`](https://huggingface.co/digitable-lol/digit-router-0.6b) | The shipping router. LoRA adapters (v1/v2/v3) over `Qwen/Qwen3-0.6B` plus merged GGUF quantisations. 424 MiB at Q5_K_M, ~500 ms per full two-step routing cycle on 8 CPU threads. |
28
  | [`digit-router-1.7b`](https://huggingface.co/digitable-lol/digit-router-1.7b) | The same training run at 1.7B. Higher routing accuracy; measurably not worth 3× the parameters for this task. |
29
  | [`digit-router-experiments`](https://huggingface.co/digitable-lol/digit-router-experiments) | Three adapters that lost — Vikhr, ruadapt, QVikhr-3. Published so the negative result stays reproducible instead of becoming folklore. |
30
+ | [`specgen-qwen3-1.7b`](https://huggingface.co/digitable-lol/specgen-qwen3-1.7b) | Writes executable FTS specifications from Russian prose. 99.4 % of its output passes the real compiler on a held-out set whose templates and subject areas it never saw; the untuned base scores 0. 133 MiB of LoRA, trained in 57 minutes inside 10 GiB. |
31
+
32
+ ### Datasets
33
+
34
+ | Repository | What it is |
35
+ |---|---|
36
+ | [`digit-router-dataset`](https://huggingface.co/datasets/digitable-lol/digit-router-dataset) | 34 709 Russian rows for the two-step router over a 95-utility catalogue. 23.8 % are refusals, and that fraction is the point. |
37
+ | [`hard-negatives-ru`](https://huggingface.co/datasets/digitable-lol/hard-negatives-ru) | 8 470 machine-verified near-miss question–chunk pairs, 2 967 paraphrases, and a 498-item twins set on which a zero-false-accept threshold was measured to cost 94 % of recall. |
38
+ | [`digit-eval-tasks`](https://huggingface.co/datasets/digitable-lol/digit-eval-tasks) | 400 Russian evaluation tasks, 250 main and 150 red-team, for a system required to answer only from a tool, a quote or a certificate. |
39
+ | [`fts-specgen-dataset`](https://huggingface.co/datasets/digitable-lol/fts-specgen-dataset) | 10 500 "Russian task → executable FTS specification" pairs, every one compiler-checked — plus a **negative control of 4 400 deliberately corrupted documents** measuring what the checker misses. |
40
 
41
  ## How to read our numbers
42
 
43
+ Three conventions run through every card, and each exists because the obvious way to
44
  report these numbers is misleading:
45
 
46
  * **A counted refusal is not a conscious refusal.** An eval harness scores an unparseable
47
  answer as a refusal, so a model that merely breaks scores like a model that knows when
48
  to decline. We always report both columns. The untuned 0.6B base scores 75.3 % counted
49
  against 9.3 % conscious — a 66-point gap that is entirely broken output.
50
+
51
+ * **A filter that never rejects anything is not evidence.** "All 10 500 documents passed
52
+ the compiler" is worth nothing until somebody shows the compiler can fail things, so
53
+ `fts-specgen-dataset` ships 4 400 documents corrupted on purpose in eleven ways and the
54
+ verdict on each. The gate catches 91.6 %; the card names the three corruptions it
55
+ catches worst and why. That measurement also *changed the work*: one class was caught
56
+ 5.0 % of the time until the generator was taught to emit boundary examples, after which
57
+ it was 72.8 %.
58
+
59
  * **Known defects are stated before the good tables, not in a footnote.** The imatrix
60
+ quantisation of the 0.6B router measurably breaks its ability to refuse and carries a
61
+ do-not-deploy warning next to the file. At Q4 the routers emit structurally flawless
62
+ tool calls with invented arguments, and a GBNF grammar does not catch that. Our v3
63
+ adapters are a routing regression against v2 on a single seed. The spec generator omits
64
+ the theorem in 127 of 1 491 otherwise-valid documents, and its own training data turns
65
+ out never to have shown it a utility and a theorem in the same document — a hole found
66
+ only after the model made it visible. All of this is on the model pages.
67
+
68
+ A fourth, specific to the verified work: **compiling is not meaning.** Our gate proves a
69
+ specification parses, type checks, runs its own examples and proves its own theorem. It
70
+ does not prove the document says what the request asked for. We have not measured that,
71
+ and we say so rather than letting a 99.4 % imply it.
72
 
73
  Every published file's sha256 is recorded in a `MANIFEST.json` in its repository. We track
74
  runs by weight hash rather than by tag, because a tag was once re-created from a different
75
  build while a 250-task evaluation was in flight.
76
 
77
+ Base models are `Qwen/Qwen3-*` under Apache-2.0. The router training data is derived from a
78
  GPL-3.0 utility catalogue; we state that provenance on every page and do not claim to have
79
+ resolved what it means for weights. The FTS work has no such question — the language, the
80
+ compiler and the gate are ours under BSD-2-Clause — and the cards say so plainly instead of
81
+ reusing the cautious wording where it is not needed.
82
 
83
  ---
84
 
 
88
  [courses.digitable.life](https://courses.digitable.life). Ограничение архитектуры
89
  необычное: языковой модели запрещено быть источником фактов. Содержание ответа даёт
90
  детерминированная утилита, дословная цитата из корпуса курсов или формальный сертификат.
91
+ Модель выбирает, что вызвать, — либо пишет артефакт, который затем проверяет программа.
92
+
93
+ **Здесь лежат маршрутизаторы и генератор спецификаций, а не универсальные ассистенты.**
94
+ Маршрутизатор относит запрос к категории инструментов и извлекает аргументы либо
95
+ отказывается; ответ он не пишет. Генератор спецификаций пишет исполняемые документы
96
+ [FTS](https://github.com/digitable-lol/flang) — и им тоже не доверяют: их компилируют,
97
+ типизируют, исполняют и доказывают настоящим инструментарием.
98
+
99
+ Три правила чтения наших чисел.
100
+
101
+ **Первое: засчитанный отказ ≠ ос��знанный** — харнесс считает отказом любой неразбираемый
102
+ ответ, поэтому сломанная модель выглядит как осторожная; мы всегда печатаем обе колонки.
103
+
104
+ **Второе: фильтр, который никогда ничего не отверг, — не доказательство.** Фраза «все
105
+ 10 500 документов прошли компилятор» ничего не стоит, пока не показано, что компилятор
106
+ вообще способен что-то отвергнуть. Поэтому в `fts-specgen-dataset` опубликованы 4 400
107
+ намеренно испорченных документов (одиннадцать видов порчи) и вердикт по каждому. Гейт
108
+ ловит 91,6 %, и в карточке названы три вида порчи, которые он ловит хуже всего, и почему.
109
+ Этот замер ещё и изменил работу: один класс ловился на 5,0 %, пока генератор не научили
110
+ добавлять граничные примеры, — после этого 72,8 %.
111
+
112
+ **Третье: известные дефекты стоят до таблиц с хорошими числами, а не в примечаниях.**
113
+ imatrix-квант 0.6B ломает способность отказываться и помечен как непригодный к поставке;
114
+ при Q4 модель выдаёт структурно безупречные вызовы с выдуманными аргументами, и грамматика
115
+ этого не ловит; адаптеры v3 — измеренный регресс против v2 на одном seed; генератор
116
+ спецификаций теряет теорему в 127 документах из 1 491 формально валидных — и его обучающие
117
+ данные, как выяснилось, **ни разу** не показали ему утилиту и теорему в одном документе.
118
+
119
+ И четвёртое, для верифицированной части: **компилируется — не значит «означает».** Гейт
120
+ доказывает, что документ разбирается, типизируется, исполняет свои примеры и доказывает
121
+ свою теорему. Что он означает ровно то, что просили словами, — не доказывает. Мы это не
122
+ измеряли и пишем об этом прямо, а не даём числу 99,4 % намекнуть на большее.
123
 
124
  sha256 каждого опубликованного файла записан в `MANIFEST.json` соответствующего
125
  репозитория.