Maggio33 commited on
Commit
270ebad
·
verified ·
1 Parent(s): aa8f70d

Card: state explicitly that no ARC data was added to pretraining (base, no fine-tune)

Browse files
Files changed (1) hide show
  1. README.md +3 -1
README.md CHANGED
@@ -10,6 +10,8 @@ datasets: [SlayerLab/gollem-v5-arcmix-9b, SlayerLab/minimal-en-corpus-5b, Huggin
10
 
11
  A small decoder-only English language model (139.3M parameters) trained from scratch by Fabryka AI. This is the **pretrained base** of [Slayer139 1.01](https://huggingface.co/SlayerLab/Slayer139-1.01) and [Slayer139 1.02](https://huggingface.co/SlayerLab/Slayer139-1.02): the end of the 8× RTX 5090 pretraining run (step 90,310), with **no fine-tuning** of any kind. Slayer139 1.01 is this model fine-tuned on ARC-Easy train; Slayer139 1.02 is this model after continued pretraining and a fine-tune. [Slayer139 1.01b](https://huggingface.co/SlayerLab/Slayer139-1.01b) is a separate run with the same recipe on a smaller machine, not this checkpoint.
12
 
 
 
13
  **Author:** Arkadiusz Słota (Fabryka AI / SlayerLab). Training, evaluation and safety gates were run by the Fabryka AI agent team under his direction.
14
 
15
  ## Result
@@ -56,7 +58,7 @@ Aggregate corpus; each source keeps its upstream terms.
56
  ## Limitations
57
 
58
  - English only; small model; one seed.
59
- - The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC test set); ARC-Easy is therefore an in-domain benchmark for this model.
60
  - Not instruction-tuned and not fine-tuned on any benchmark.
61
 
62
  ## OpenStax attribution
 
10
 
11
  A small decoder-only English language model (139.3M parameters) trained from scratch by Fabryka AI. This is the **pretrained base** of [Slayer139 1.01](https://huggingface.co/SlayerLab/Slayer139-1.01) and [Slayer139 1.02](https://huggingface.co/SlayerLab/Slayer139-1.02): the end of the 8× RTX 5090 pretraining run (step 90,310), with **no fine-tuning** of any kind. Slayer139 1.01 is this model fine-tuned on ARC-Easy train; Slayer139 1.02 is this model after continued pretraining and a fine-tune. [Slayer139 1.01b](https://huggingface.co/SlayerLab/Slayer139-1.01b) is a separate run with the same recipe on a smaller machine, not this checkpoint.
12
 
13
+ **Base model, no ARC fine-tuning.** No ARC dataset (questions or answers, any split) was added to the pretraining data, and this base was never fine-tuned. ARC-Easy train was used only later, in the separate fine-tunes that produced Slayer139 1.01 and 1.02. The pretraining mix does give extra weight to web documents on science topics related to ARC (see Training data and Limitations); web documents overlapping ARC validation/test were removed.
14
+
15
  **Author:** Arkadiusz Słota (Fabryka AI / SlayerLab). Training, evaluation and safety gates were run by the Fabryka AI agent team under his direction.
16
 
17
  ## Result
 
58
  ## Limitations
59
 
60
  - English only; small model; one seed.
61
+ - The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC validation and test sets; no ARC questions were added); ARC-Easy is therefore an in-domain benchmark for this model.
62
  - Not instruction-tuned and not fine-tuned on any benchmark.
63
 
64
  ## OpenStax attribution