zils-lite-body-20: the mean of 3 LoRA runs (factor 1) on the d3-lite soup 0db42586, trained on labelled rows and on d3-ultra teacher probabilities

This is a decision body for the Zils d3-lite soup pool (Bittensor subnet 127), Qwen/Qwen3.5-0.8B family.

Body

The body's model.safetensors sha256 is 33c88d4019369afa029cd1c4e1c3b6af344c174a0a75f9ff6be900cd7abefb88; its Zils body digest is bd1f463f4f3b55ec27fa17fc0d98956ce9a0322a4575532aa694af9c6bace667. It is START + sum over the 3 runs of c x (RUN - START), on every tensor, with c = 0.333333333333 for each run (our weight-difference tool delta_combo.py v1.4, sha256 prefix 0a9b49b1): 1 x the mean of the 3 runs' changes. 114 tensors differ from the start. No other body is merged.

Start (S11)

S11 is the official soup of the d3-lite round d3-lite-2026-11-14 (soup hash 0db42586f066681c5d76dab8b61430cba05996daddaeb43054eb7d56f1a05f5b), rebuilt by us with the official Zils soup code as the uniform average of its 2 public member bodies at the revisions their owners had committed:

The rebuilt soup's hash equals the published one. The start's Zils body digest is 3706cc7dc275f9f99fd82fb18de00dc2be237fb1240124c81287b0c238fbe171 (weights file sha256 7adf66b342190d5ce37ecfe76be6a95153136f135acd849ef7da47124a6900e1).

Training (3 runs, seeds 5201, 5211, 5221, one recipe)

Each run starts from S11 and uses our trainer train_soft_c_pin_v389.py (sha256 prefix 61e6cb93). The values below are the flags of our lane file (sha256 prefix e73991ce) and wrapper (sha256 prefix d2184ae3), read back from each run's train_meta.json and receipt line.

  • LoRA adapters (rank 16, alpha 32, dropout 0.05) on 114 modules (all layers), merged into the full weights after training (start + alpha/rank x B A, stored bf16)
  • AdamW, learning rate 1e-4, weight decay 0, 50 warm-up steps then cosine decay; 1 epoch; batch 8 with 2 accumulation steps; rows up to 2,048 tokens; 0 rows and 0 pins skipped
  • loss per row: on the 49,997 teacher rows, KL(teacher || body) with weight 1 at temperature 1; on the 27,380 labelled rows, cross-entropy on the row's label with weight 1 (label smoothing 0, Brier 0, RPS 0); on the 12,095 pin prompts, KL to the start's own answer distribution, computed live from the start weights, with weight 1.0
  • run B1Ls5201 (seed 5201): 77,377 rows, 12,095 pins, 197.1 min, final loss 0.2800
  • run B1Ls5211 (seed 5211): 77,377 rows, 12,095 pins, 194.8 min, final loss 0.2932
  • run B1Ls5221 (seed 5221): 77,377 rows, 12,095 pins, 197.8 min, final loss 0.3148

Training data

One training file of 77,377 rows (sha256 8616785c0d7860ad9eb296a51e7f6b01883b7fe3b2a27bfb27e473611943347b) and one file of 12,095 pin prompts (sha256 f682ca66f8aead00b61d0e143cdd17ae7d313789ab1b18a69aa30ba67246f6c1):

category labelled rows (choice / yes-no / score) teacher rows (choice / yes-no / score) pin prompts (choice / yes-no / score)
reasoning 2,509 / 2,510 / 2,510 20,433 / 183 / 0 3,511 / 1,232 / 144
language 2,590 / 2,590 / 2,153 12,045 / 1,073 / 583 1,996 / 1,075 / 130
retrieval 1,661 / 1,661 / 1,840 6,229 / 122 / 41 1,084 / 625 / 71
tools 1,590 / 1,590 / 1,341 1,156 / 0 / 0 182 / 161 / 3
arts 939 / 958 / 938 7,586 / 335 / 211 1,106 / 688 / 87
total 27,380 49,997 12,095

Labelled rows (27,380)

Where each row's label comes from, counted from the row's provenance record (our provenance file ND30K_PROVENANCE, joined by row id; all 27,380 rows found):

label origin rows share
from a public dataset's TRAIN-split record (the dataset's label, or a value our converter derived from the record), put into the exam layout by our code 15,749 57.5%
from a public dataset's TRAIN-split record (the dataset's label, or a value derived by code from the record), from our earlier row pool 900 3.3%
derived by our code from a public dataset record (CMU Pronouncing Dictionary) 475 1.7%
computed by our own program generators from the item's own specification 4,450 16.3%
the writer's own answer: written by 3 Claude sessions of our team (3,279, 2,272 and 249 rows) 5,800 21.2%
the writer's own answer: written by local models (Phi-4 5, Mistral-Nemo 1) 6 0.02%

The 15,749 rows of the first kind came through 20 source groups. For each group a label-free sample (60 rows for 19 groups, 43 rows for 1 group) was answered blind, and the group passed: at most 6 answers disagreed with the row's label. The 4 groups that failed this check (goemo, poem, siqa, snli) gave no rows by this route. The written rows carry no human label.

source rows licence (as recorded in our data manifest)
our earlier row pool: written by a Claude session of our team 3,279 our own writing; a public TRAIN row served as style (or topic and style) example only
nyu-mll/glue 3,206 MNLI: OANC + CC BY-SA 3.0 / CC BY 3.0 fiction parts; QNLI: CC BY-SA 4.0 (from SQuAD)
our earlier row pool: written by a Claude session of our team 2,272 our own writing; a public TRAIN row served as style (or topic and style) example only
baber/piqa 1,805 AFL-3.0
nvidia/When2Call 1,466 CC BY 4.0
openai/gsm8k 1,295 MIT
dstc8-schema-guided-dialogue 1,205 CC BY-SA 4.0
GEN nd_gen_lang_score_k6 1,189 own code (program generator)
google-research-datasets/paws 1,048 PAWS licence (any purpose)
allenai/ai2_arc 1,018 CC BY-SA 4.0
allenai/cosmos_qa 987 CC BY 4.0
rajpurkar/squad_v2 922 CC BY-SA 4.0
fancyzhx/dbpedia_14 853 CC BY-SA 3.0
GEN gen_rule_lang_var_score_v0 543 own code (program generator)
GEN gen_rule_retrieval_hard_v0 532 own code (program generator)
google/boolq 526 CC BY-SA 3.0
iSarcasmEval 520 MIT
nightingal3/fig-qa 497 MIT
jc4p/CMUdict 475 BSD 2-Clause (CMU Pronouncing Dictionary)
GEN gen_rule_retrieval_whole_v0 416 own code (program generator)
GEN nd_gen_reason_k6 314 own code (program generator)
GEN nd_gen_tools_policy 306 own code (program generator)
GEN gen_rule_arts_var_score_v1 266 own code (program generator)
our earlier row pool: written by a Claude session of our team 249 our own writing; a public TRAIN row served as style (or topic and style) example only
GEN gen_rule_arts_var_choice_v1 244 own code (program generator)
GEN gen_reasoning_score_v3 239 own code (program generator)
allenai/quoref 182 CC BY 4.0
toolace 179 Apache-2.0
our earlier row pool (iSarcasmEval task A) 147 MIT
our earlier row pool (nyu-mll/glue mnli) 129 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (baber/piqa) 112 PIQA (AFL-3.0)
GEN gen_rule_arts_var_noul_v1 89 own code (program generator)
our earlier row pool (ft_mnli_choice_r2) 73 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (ft_piqa_noul_r3) 68 PIQA (AFL-3.0)
our earlier row pool (ft_qnli_noul) 63 QNLI (from SQuAD, CC BY-SA 4.0)
GEN gen_rule_retrieval_choice/join_choice_v0 62 own code (program generator)
GEN nd_gen_lang_s1 60 own code (program generator)
GEN gen_rule_retrieval_score_v0 60 own code (program generator)
our earlier row pool (ft_mnli_choice_r3) 59 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (code:gen_reasoning_score_v3) 57 generated by code (team tool); own code
GEN gen_rule_retrieval_noul/join_noul_v0 50 own code (program generator)
our earlier row pool (ft_squad_sent_noul) 47 SQuAD v2 (CC BY-SA 4.0)
hermes-fc 40 Apache-2.0
our earlier row pool (ft_piqa_noul) 39 PIQA (AFL-3.0)
our earlier row pool (ft_mnli_choice) 37 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (nyu-mll/glue qnli) 37 QNLI (from SQuAD, CC BY-SA 4.0)
our earlier row pool (ft_piqa_noul_r2) 31 PIQA (AFL-3.0)
our earlier row pool (code:gen_rule_retrieval_score_v0) 23 own code
our earlier row pool (ft_isarcasm_noul) 19 MIT
our earlier row pool (ft_qnli_noul_r2) 17 QNLI (from SQuAD, CC BY-SA 4.0)
our earlier row pool (ft_snli_choice) 15 SNLI (CC BY-SA 4.0)
our earlier row pool (ft_mnli_noul) 7 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool: written by local models (Phi-4, Mistral-Nemo) 6 our own writing; a public TRAIN row served as style example only

Teacher rows (49,997)

Each teacher row carries option probabilities from vllm-sr/d3-ultra (a public 27B model, Apache-2.0; not a Zils miner's body), as published in zils-ai/zils-relay-answers at revision 80c4668a53c1c3a9d0deb7c3c89dd08df30ae96c (dataset card licence apache-2.0). They have no gold label. Each row's question text comes from a public source dataset that the row names with its file and row index (2 datasets, 249 source names; splits: train 49,997; recorded licence use: commercial 49,997).

Licences recorded in the teacher rows (several values per row where the dataset lists several):

recorded licence rows
mit 10,168
cc-by-4.0 9,018
apache-2.0 8,758
cc-by-sa-4.0 2,465
cc-by-sa-3.0 2,290
CC BY 4.0 (DPI) 2,282
cc0-1.0 1,911
apache-2.0, Apache License 2.0 (DPI) 1,579
cc-by-4.0, CC BY 4.0 (DPI), Custom (DPI) 1,550
bsd 1,272
CC0 1.0 (DPI) 947
cc-by-4.0, CC BY 4.0 (DPI) 815
apache-2.0, Custom (DPI) 810
apache-2.0, apache-2.0 (original source) 646
mit, Custom (DPI) 567
afl-3.0 561
gpl 508
odc-by 502
cc-by-sa-4.0, CC BY-SA 4.0 (DPI) 472
cc, CC BY-SA 4.0 (DPI) 461
apache-2.0, MIT License (DPI) 336
cc-by-4.0, cc-by-4.0 (original source) 325
cc-by-sa-4.0, CC BY 4.0 (DPI) 291
Apache License 2.0 (DPI), CC BY-SA 4.0 (DPI) 273
mit, MIT License (DPI) 191
gpl, Custom (DPI) 142
gpl-3.0 125
cc-by-sa-3.0, CC BY-SA 3.0 (DPI) 120
CC BY-SA 4.0 (DPI) 114
BSD 2-Clause License (DPI) 112
other, cdla-permissive-1.0 (original source) 101
cc0-1.0, CC0 1.0 (DPI) 87
cc-by-3.0 57
cc-by-4.0, Custom (DPI) 43
cc-by-sa-4.0, cc-by-sa-4.0 (original source) 36
apache-2.0 (original source) 29
mit, mit (original source) 22
odbl 11

Pin prompts (12,095)

Prompts only, from public source datasets (219 source names, 218 of them also among the teacher rows' sources; splits: train 12,095). None of them share their exact state with a training row. Their only target is the start's own answer distribution, computed live; they carry no label and no probabilities.

Evaluation

These are our own reads, not an official score.

  • Copy screen (the official sketch method, 3 seeds, the current soup kept as the soup; 87 field bodies): no copy found. Cosine to the soup at most 0.829325 (Zils line 0.999). Against our other bodies: cosine at most 0.7995, share explained at most 0.000 (line 0.30). Against the field: cosine at most 0.8137, share explained at most 0.021. Behaviour distance to the soup 0.0267 bits, to our other bodies at least 0.0326 bits (Zils twin line 0.005; ours 0.01).
  • Our reads against the soup (our rebuilt start, same reads for both):
    • broad dev read (3,279 items): accuracy change +1.28 points (standard error 0.50)
    • hold-out read (our sealed hold-out rows, 150 rows): mean change in the probability of the gold answer +0.0506
    • exam-style read (C3, 10 registered splits): mean -1.57 points, 0 of 10 splits above 0
    • per-family read: lowest family mean -13.13 points (language-known-noul-k2; our line -3.0)
    • tools read (our dev tools items per form; multi-class Brier, lower is better, body vs soup): k2 0.102 vs 0.119 (acc 93.3% vs 93.3%, n 60); k6 0.052 vs 0.064 (acc 97.1% vs 96.2%, n 240); w2c 0.459 vs 0.574 (acc 68.5% vs 64.5%, n 200)
    • own change (the part of the body's change that the soup, the house models and 14 known Lite models do not explain; 3 sketch seeds): mean 0.3194, lowest seed 0.3116; highest cosine to a known model 0.479
  • Our pick rule (copy screen; then the dev, hold-out and tools reads not below the soup; then no family below -3.0; then the largest own change): this body does not pass it: per-family read (below our line).

Licence

Apache-2.0 (LICENSE). NOTICE lists the start, the teacher and the data sources with their licences. The training rows are not distributed with this body. Nothing here is legal advice; check the original datasets' terms for your use.

Downloads last month
32
Safetensors
Model size
0.9B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for tensor-tailor/zils-lite-body-20

Finetuned
(2)
this model