Card: state explicitly that no ARC data was added to pretraining (base, no fine-tune)
Browse files
README.md
CHANGED
|
@@ -10,6 +10,8 @@ datasets: [SlayerLab/gollem-v5-arcmix-9b, SlayerLab/minimal-en-corpus-5b, Huggin
|
|
| 10 |
|
| 11 |
A small decoder-only English language model (139.3M parameters) trained from scratch by Fabryka AI. This is the **pretrained base** of [Slayer139 1.01](https://huggingface.co/SlayerLab/Slayer139-1.01) and [Slayer139 1.02](https://huggingface.co/SlayerLab/Slayer139-1.02): the end of the 8× RTX 5090 pretraining run (step 90,310), with **no fine-tuning** of any kind. Slayer139 1.01 is this model fine-tuned on ARC-Easy train; Slayer139 1.02 is this model after continued pretraining and a fine-tune. [Slayer139 1.01b](https://huggingface.co/SlayerLab/Slayer139-1.01b) is a separate run with the same recipe on a smaller machine, not this checkpoint.
|
| 12 |
|
|
|
|
|
|
|
| 13 |
**Author:** Arkadiusz Słota (Fabryka AI / SlayerLab). Training, evaluation and safety gates were run by the Fabryka AI agent team under his direction.
|
| 14 |
|
| 15 |
## Result
|
|
@@ -56,7 +58,7 @@ Aggregate corpus; each source keeps its upstream terms.
|
|
| 56 |
## Limitations
|
| 57 |
|
| 58 |
- English only; small model; one seed.
|
| 59 |
-
- The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC test
|
| 60 |
- Not instruction-tuned and not fine-tuned on any benchmark.
|
| 61 |
|
| 62 |
## OpenStax attribution
|
|
|
|
| 10 |
|
| 11 |
A small decoder-only English language model (139.3M parameters) trained from scratch by Fabryka AI. This is the **pretrained base** of [Slayer139 1.01](https://huggingface.co/SlayerLab/Slayer139-1.01) and [Slayer139 1.02](https://huggingface.co/SlayerLab/Slayer139-1.02): the end of the 8× RTX 5090 pretraining run (step 90,310), with **no fine-tuning** of any kind. Slayer139 1.01 is this model fine-tuned on ARC-Easy train; Slayer139 1.02 is this model after continued pretraining and a fine-tune. [Slayer139 1.01b](https://huggingface.co/SlayerLab/Slayer139-1.01b) is a separate run with the same recipe on a smaller machine, not this checkpoint.
|
| 12 |
|
| 13 |
+
**Base model, no ARC fine-tuning.** No ARC dataset (questions or answers, any split) was added to the pretraining data, and this base was never fine-tuned. ARC-Easy train was used only later, in the separate fine-tunes that produced Slayer139 1.01 and 1.02. The pretraining mix does give extra weight to web documents on science topics related to ARC (see Training data and Limitations); web documents overlapping ARC validation/test were removed.
|
| 14 |
+
|
| 15 |
**Author:** Arkadiusz Słota (Fabryka AI / SlayerLab). Training, evaluation and safety gates were run by the Fabryka AI agent team under his direction.
|
| 16 |
|
| 17 |
## Result
|
|
|
|
| 58 |
## Limitations
|
| 59 |
|
| 60 |
- English only; small model; one seed.
|
| 61 |
+
- The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC validation and test sets; no ARC questions were added); ARC-Easy is therefore an in-domain benchmark for this model.
|
| 62 |
- Not instruction-tuned and not fine-tuned on any benchmark.
|
| 63 |
|
| 64 |
## OpenStax attribution
|