zils-mini-body-36: the mean of 2 LoRA runs (factor 1) on the d3-mini soup 4e9fcfd2, trained on labelled rows and on d3-ultra teacher probabilities

This is a full body for the Zils d3-mini pool (Bittensor subnet 127); it uses the model d3-mini-4.5b-v1 from the Zils repository at revision e9e87b3e.

Body

The body's model.safetensors sha256 is d2c84660dadaa0294e3113a8849ad66791f59ba10ff32050052e3aa52db77292; its Zils body digest is a8274188b05d5a527daf4052ea5b6ece4eff3253b5881170e5131e957b554e83. It is START + sum over the 2 runs of c x (RUN - START), on every tensor, with c = 0.5 for each run (our weight-difference tool delta_combo.py v1.4, sha256 prefix 0a9b49b1): 1 x the mean of the 2 runs' changes. 152 tensors differ from the start. No other body is merged.

Start (MS14)

MS14 is the official soup of the d3-mini round d3-mini-2026-11-14 (soup hash 4e9fcfd2642f6e7ee1c9d24bd362b388f85a5771af8a49c215e3794c3c2468c8), rebuilt by us with the official Zils soup code as the uniform average of its 3 public member bodies at the revisions their owners had committed:

The rebuilt soup's hash equals the published one. The start's Zils body digest is 67f591041ab9b24efb241d3429ee0a7366b95ae7759293a812cb7d09a2bda88f.

Training (2 runs, seeds 7201, 7211, one recipe)

Each run starts from MS14 and uses our trainer train_soft_c_pin_v389.py (sha256 prefix 61e6cb93). The values below are the flags of our lane file (sha256 prefix eaabffea) and wrapper (sha256 prefix 91d9682a), read back from each run's train_meta.json and receipt line.

  • LoRA adapters (rank 32, alpha 64, dropout 0.05) on 152 modules (all layers), merged into the full weights after training (start + alpha/rank x B A, stored bf16)
  • AdamW, learning rate 1e-4, weight decay 0, 50 warm-up steps then cosine decay; 1 epoch; batch 8 with 2 accumulation steps; rows up to 2,048 tokens; 0 rows and 0 pins skipped
  • loss per row: on the 49,997 teacher rows, KL(teacher || body) with weight 1 at temperature 1; on the 27,380 labelled rows, cross-entropy on the row's label with weight 1 (label smoothing 0, Brier 0, RPS 0); on the 12,095 pin prompts, KL to the start's own answer distribution, computed live from the start weights, with weight 1.0
  • run B1Ms7201 (seed 7201): 77,377 rows, 12,095 pins, 208.5 min, final loss 0.1583
  • run B1Ms7211 (seed 7211): 77,377 rows, 12,095 pins, 209.4 min, final loss 0.1537

Training data

One training file of 77,377 rows (sha256 8616785c0d7860ad9eb296a51e7f6b01883b7fe3b2a27bfb27e473611943347b) and one file of 12,095 pin prompts (sha256 f682ca66f8aead00b61d0e143cdd17ae7d313789ab1b18a69aa30ba67246f6c1):

category labelled rows (choice / yes-no / score) teacher rows (choice / yes-no / score) pin prompts (choice / yes-no / score)
reasoning 2,509 / 2,510 / 2,510 20,433 / 183 / 0 3,511 / 1,232 / 144
language 2,590 / 2,590 / 2,153 12,045 / 1,073 / 583 1,996 / 1,075 / 130
retrieval 1,661 / 1,661 / 1,840 6,229 / 122 / 41 1,084 / 625 / 71
tools 1,590 / 1,590 / 1,341 1,156 / 0 / 0 182 / 161 / 3
arts 939 / 958 / 938 7,586 / 335 / 211 1,106 / 688 / 87
total 27,380 49,997 12,095

Labelled rows (27,380)

Where each row's label comes from, counted from the row's provenance record (our provenance file ND30K_PROVENANCE, joined by row id; all 27,380 rows found):

label origin rows share
from a public dataset's TRAIN-split record (the dataset's label, or a value our converter derived from the record), put into the exam layout by our code 15,749 57.5%
from a public dataset's TRAIN-split record (the dataset's label, or a value derived by code from the record), from our earlier row pool 900 3.3%
derived by our code from a public dataset record (CMU Pronouncing Dictionary) 475 1.7%
computed by our own program generators from the item's own specification 4,450 16.3%
the writer's own answer: written by 3 Claude sessions of our team (3,279, 2,272 and 249 rows) 5,800 21.2%
the writer's own answer: written by local models (Phi-4 5, Mistral-Nemo 1) 6 0.02%

The 15,749 rows of the first kind came through 20 source groups. For each group a label-free sample (60 rows for 19 groups, 43 rows for 1 group) was answered blind, and the group passed: at most 6 answers disagreed with the row's label. The 4 groups that failed this check (goemo, poem, siqa, snli) gave no rows by this route. The written rows carry no human label.

source rows licence (as recorded in our data manifest)
our earlier row pool: written by a Claude session of our team 3,279 our own writing; a public TRAIN row served as style (or topic and style) example only
nyu-mll/glue 3,206 MNLI: OANC + CC BY-SA 3.0 / CC BY 3.0 fiction parts; QNLI: CC BY-SA 4.0 (from SQuAD)
our earlier row pool: written by a Claude session of our team 2,272 our own writing; a public TRAIN row served as style (or topic and style) example only
baber/piqa 1,805 AFL-3.0
nvidia/When2Call 1,466 CC BY 4.0
openai/gsm8k 1,295 MIT
dstc8-schema-guided-dialogue 1,205 CC BY-SA 4.0
GEN nd_gen_lang_score_k6 1,189 own code (program generator)
google-research-datasets/paws 1,048 PAWS licence (any purpose)
allenai/ai2_arc 1,018 CC BY-SA 4.0
allenai/cosmos_qa 987 CC BY 4.0
rajpurkar/squad_v2 922 CC BY-SA 4.0
fancyzhx/dbpedia_14 853 CC BY-SA 3.0
GEN gen_rule_lang_var_score_v0 543 own code (program generator)
GEN gen_rule_retrieval_hard_v0 532 own code (program generator)
google/boolq 526 CC BY-SA 3.0
iSarcasmEval 520 MIT
nightingal3/fig-qa 497 MIT
jc4p/CMUdict 475 BSD 2-Clause (CMU Pronouncing Dictionary)
GEN gen_rule_retrieval_whole_v0 416 own code (program generator)
GEN nd_gen_reason_k6 314 own code (program generator)
GEN nd_gen_tools_policy 306 own code (program generator)
GEN gen_rule_arts_var_score_v1 266 own code (program generator)
our earlier row pool: written by a Claude session of our team 249 our own writing; a public TRAIN row served as style (or topic and style) example only
GEN gen_rule_arts_var_choice_v1 244 own code (program generator)
GEN gen_reasoning_score_v3 239 own code (program generator)
allenai/quoref 182 CC BY 4.0
toolace 179 Apache-2.0
our earlier row pool (iSarcasmEval task A) 147 MIT
our earlier row pool (nyu-mll/glue mnli) 129 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (baber/piqa) 112 PIQA (AFL-3.0)
GEN gen_rule_arts_var_noul_v1 89 own code (program generator)
our earlier row pool (ft_mnli_choice_r2) 73 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (ft_piqa_noul_r3) 68 PIQA (AFL-3.0)
our earlier row pool (ft_qnli_noul) 63 QNLI (from SQuAD, CC BY-SA 4.0)
GEN gen_rule_retrieval_choice/join_choice_v0 62 own code (program generator)
GEN nd_gen_lang_s1 60 own code (program generator)
GEN gen_rule_retrieval_score_v0 60 own code (program generator)
our earlier row pool (ft_mnli_choice_r3) 59 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (code:gen_reasoning_score_v3) 57 generated by code (team tool); own code
GEN gen_rule_retrieval_noul/join_noul_v0 50 own code (program generator)
our earlier row pool (ft_squad_sent_noul) 47 SQuAD v2 (CC BY-SA 4.0)
hermes-fc 40 Apache-2.0
our earlier row pool (ft_piqa_noul) 39 PIQA (AFL-3.0)
our earlier row pool (ft_mnli_choice) 37 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool (nyu-mll/glue qnli) 37 QNLI (from SQuAD, CC BY-SA 4.0)
our earlier row pool (ft_piqa_noul_r2) 31 PIQA (AFL-3.0)
our earlier row pool (code:gen_rule_retrieval_score_v0) 23 own code
our earlier row pool (ft_isarcasm_noul) 19 MIT
our earlier row pool (ft_qnli_noul_r2) 17 QNLI (from SQuAD, CC BY-SA 4.0)
our earlier row pool (ft_snli_choice) 15 SNLI (CC BY-SA 4.0)
our earlier row pool (ft_mnli_noul) 7 MNLI (OANC / mixed open licences, GLUE)
our earlier row pool: written by local models (Phi-4, Mistral-Nemo) 6 our own writing; a public TRAIN row served as style example only

Teacher rows (49,997)

Each teacher row carries option probabilities from vllm-sr/d3-ultra (a public 27B model, Apache-2.0; not a Zils miner's body), as published in zils-ai/zils-relay-answers at revision 80c4668a53c1c3a9d0deb7c3c89dd08df30ae96c (dataset card licence apache-2.0). They have no gold label. Each row's question text comes from a public source dataset that the row names with its file and row index (2 datasets, 249 source names; splits: train 49,997; recorded licence use: commercial 49,997).

Licences recorded in the teacher rows (several values per row where the dataset lists several):

recorded licence rows
mit 10,168
cc-by-4.0 9,018
apache-2.0 8,758
cc-by-sa-4.0 2,465
cc-by-sa-3.0 2,290
CC BY 4.0 (DPI) 2,282
cc0-1.0 1,911
apache-2.0, Apache License 2.0 (DPI) 1,579
cc-by-4.0, CC BY 4.0 (DPI), Custom (DPI) 1,550
bsd 1,272
CC0 1.0 (DPI) 947
cc-by-4.0, CC BY 4.0 (DPI) 815
apache-2.0, Custom (DPI) 810
apache-2.0, apache-2.0 (original source) 646
mit, Custom (DPI) 567
afl-3.0 561
gpl 508
odc-by 502
cc-by-sa-4.0, CC BY-SA 4.0 (DPI) 472
cc, CC BY-SA 4.0 (DPI) 461
apache-2.0, MIT License (DPI) 336
cc-by-4.0, cc-by-4.0 (original source) 325
cc-by-sa-4.0, CC BY 4.0 (DPI) 291
Apache License 2.0 (DPI), CC BY-SA 4.0 (DPI) 273
mit, MIT License (DPI) 191
gpl, Custom (DPI) 142
gpl-3.0 125
cc-by-sa-3.0, CC BY-SA 3.0 (DPI) 120
CC BY-SA 4.0 (DPI) 114
BSD 2-Clause License (DPI) 112
other, cdla-permissive-1.0 (original source) 101
cc0-1.0, CC0 1.0 (DPI) 87
cc-by-3.0 57
cc-by-4.0, Custom (DPI) 43
cc-by-sa-4.0, cc-by-sa-4.0 (original source) 36
apache-2.0 (original source) 29
mit, mit (original source) 22
odbl 11

Pin prompts (12,095)

Prompts only, from public source datasets (219 source names, 218 of them also among the teacher rows' sources; splits: train 12,095). None of them share their exact state with a training row. Their only target is the start's own answer distribution, computed live; they carry no label and no probabilities.

Evaluation

These are our own reads, not an official score.

  • Simple test (our read of 1,488 rows: the probability of the gold answer for each row, paired with MS14's own read of the same rows; the change in mean log p(gold) with a 95% bootstrap interval, 2,000 resamples):

    set rows p(gold), body vs MS14 accuracy, body vs MS14 change in mean log p(gold) [95%] read
    our MINI hold-out rows 149 0.935 vs 0.925 0.953 vs 0.966 -0.047 [-0.151, +0.033] not different
    C3 base (dev_gold + pool_known) 1,339 0.893 vs 0.887 0.926 vs 0.931 -0.009 [-0.024, +0.006] not different
    dev_gold 787 0.916 vs 0.909 0.944 vs 0.947 -0.017 [-0.039, +0.004] not different
    pool_known 552 0.860 vs 0.855 0.900 vs 0.909 +0.004 [-0.015, +0.022] not different

    The test's rule: a body is "not below MS14" when neither the hold-out rows nor the C3 base is below. Neither is below, so this body reads not below MS14.

  • Copy screen (the official sketch method, 1 seed, the current soup kept as the soup; 67 field bodies): no copy found. Cosine to the soup at most 0.587010 (Zils line 0.999). Against our other bodies: cosine at most 0.4152, share explained at most 0.0011 (line 0.30). Against the field: cosine at most 0.5587, share explained at most 0.0038. Behaviour distance to the soup 0.0231 bits, to our other bodies at least 0.0354 bits (Zils twin line 0.005; ours 0.01).

Licence

Apache-2.0 (LICENSE). NOTICE lists the start, the teacher and the data sources with their licences. The training rows are not distributed with this body. Nothing here is legal advice; check the original datasets' terms for your use.

Downloads last month
21
Safetensors
Model size
5B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for tensor-tailor/zils-mini-body-36