Hugging Face's logo

tohio
/
slm-125m-instruct

Text Generation
Safetensors
English
slm
causal-lm
decoder-only
custom-architecture
rope
gqa
swiglu
instruct
conversational
custom_code
Model card Files
xet
 Community
tohio commited on
Commit
d8ff2d9
·
verified ·
1 Parent(s): faa4a19

Export slm-125m-instruct (125.3M params)

Browse files
Files changed (6) hide show
  1. README.md +29 -29
  2. code_completion_training_metadata.json +14 -0
  3. config.json +1 -1
  4. generation_config.json +2 -8
  5. model.safetensors +2 -2
  6. tokenizer/tokenizer_config.json +1 -0
README.md CHANGED
@@ -50,29 +50,29 @@ Use [`tohio/slm-125m`](https://huggingface.co/tohio/slm-125m) for the raw base m
50
 
51
  **Pretraining corpus** — 10B curation target blended across the following sources:
52
 
53
- | Source | Target Share | Realized Share | Link |
54
- |---|---|---|---|
55
- | `common_crawl` | 5.0% | 5.00% | [Common Crawl](https://commoncrawl.org) |
56
- | `fineweb` | 10.0% | 10.00% | [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) |
57
- | `fineweb_edu` | 31.5% | 31.50% | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) |
58
- | `wikipedia` | 10.0% | 10.00% | [Wikipedia (EN)](https://huggingface.co/datasets/wikimedia/wikipedia) |
59
- | `pg19` | 2.5% | 2.50% | [PG-19 (Project Gutenberg)](https://huggingface.co/datasets/pg19) |
60
- | `pes2o` | 5.0% | 5.00% | [peS2o (academic papers)](https://huggingface.co/datasets/allenai/peS2o) |
61
- | `nemotron_cc_math` | 7.0% | 7.00% | [Nemotron CC Math](https://huggingface.co/datasets/nvidia/Nemotron-CC-Math-v1) |
62
- | `stackexchange` | 1.0% | 1.00% | [StackExchange](https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences) |
63
- | `synthetic_arithmetic` | 0.1% | 0.15% | [Synthetic arithmetic](https://huggingface.co/datasets/tohio/slm-synthetic-arithmetic) |
64
- | `synthetic_task_code` | 0.4% | 0.39% | [Synthetic task code](https://huggingface.co/datasets/tohio/slm-synthetic-task-code) |
65
- | `educational_qa_mcq_math` | 0.1% | 0.15% | [Educational QA/MCQ (math)](https://huggingface.co/datasets/tohio/slm-synthetic-educational-qa-mcq-math) |
66
- | `educational_qa_mcq_general` | 0.2% | 0.25% | [Educational QA/MCQ (general)](https://huggingface.co/datasets/tohio/slm-synthetic-educational-qa-mcq-general) |
67
- | `factual_restraint` | 0.1% | 0.07% | [Factual restraint](https://huggingface.co/datasets/tohio/slm-synthetic-factual-restraint) |
68
- | `nemotron_specialized` | 12.0% | 12.00% | [Nemotron Specialized](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Specialized-v1.1) |
69
- | `stack_v1` | 12.45% | 12.53% | [The Stack v1 dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) |
70
- | `codesearchnet` | 2.25% | 2.25% | [CodeSearchNet](https://huggingface.co/datasets/code-search-net/code_search_net) |
71
- | `stack_smol` | 0.15% | 0.09% | [The Stack (smol)](https://huggingface.co/datasets/bigcode/the-stack-smol) |
72
- | `jupyter` | 0.07% | 0.06% | [Jupyter notebooks](https://huggingface.co/datasets/bigcode/jupyter-parsed) |
73
- | `conala` | 0.07% | 0.08% | [CoNaLa](https://huggingface.co/datasets/neulab/conala) |
74
-
75
- _Realized: ~10.00B tokens (15,202,249 train + 76,393 val docs). Supply-bound sources route their deficit to FineWeb._
76
 
77
  **Fine-tuning**
78
 
@@ -91,12 +91,12 @@ Evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluat
91
 
92
  | Benchmark | Few-shot | Metric | Score |
93
  |---|---|---|---|
94
- | HellaSwag | 10-shot | acc_norm | 0.3106 |
95
- | ARC-Easy | 25-shot | acc_norm | 0.4870 |
96
- | ARC-Challenge | 25-shot | acc_norm | 0.2526 |
97
- | MMLU | 5-shot | acc | 0.2495 |
98
- | TruthfulQA | 0-shot | acc | 0.4258 |
99
- | HumanEval | 0-shot | pass@1 | 0.0000 |
100
 
101
 
102
  ## Usage
 
50
 
51
  **Pretraining corpus** — 10B curation target blended across the following sources:
52
 
53
+ | Source | Target Share | Link |
54
+ |---|---|---|
55
+ | `common_crawl` | 5.0% | [Common Crawl](https://commoncrawl.org) |
56
+ | `fineweb` | 10.0% | [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) |
57
+ | `fineweb_edu` | 31.5% | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) |
58
+ | `wikipedia` | 10.0% | [Wikipedia (EN)](https://huggingface.co/datasets/wikimedia/wikipedia) |
59
+ | `pg19` | 2.5% | [PG-19 (Project Gutenberg)](https://huggingface.co/datasets/pg19) |
60
+ | `pes2o` | 5.0% | [peS2o (academic papers)](https://huggingface.co/datasets/allenai/peS2o) |
61
+ | `nemotron_cc_math` | 7.0% | [Nemotron CC Math](https://huggingface.co/datasets/nvidia/Nemotron-CC-Math-v1) |
62
+ | `stackexchange` | 1.0% | [StackExchange](https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences) |
63
+ | `synthetic_arithmetic` | 0.1% | [Synthetic arithmetic](https://huggingface.co/datasets/tohio/slm-synthetic-arithmetic) |
64
+ | `synthetic_task_code` | 0.4% | [Synthetic task code](https://huggingface.co/datasets/tohio/slm-synthetic-task-code) |
65
+ | `educational_qa_mcq_math` | 0.1% | [Educational QA/MCQ (math)](https://huggingface.co/datasets/tohio/slm-synthetic-educational-qa-mcq-math) |
66
+ | `educational_qa_mcq_general` | 0.2% | [Educational QA/MCQ (general)](https://huggingface.co/datasets/tohio/slm-synthetic-educational-qa-mcq-general) |
67
+ | `factual_restraint` | 0.1% | [Factual restraint](https://huggingface.co/datasets/tohio/slm-synthetic-factual-restraint) |
68
+ | `nemotron_specialized` | 12.0% | [Nemotron Specialized](https://huggingface.co/datasets/nvidia/Nemotron-Pretraining-Specialized-v1.1) |
69
+ | `stack_v1` | 12.45% | [The Stack v1 dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) |
70
+ | `codesearchnet` | 2.25% | [CodeSearchNet](https://huggingface.co/datasets/code-search-net/code_search_net) |
71
+ | `stack_smol` | 0.15% | [The Stack (smol)](https://huggingface.co/datasets/bigcode/the-stack-smol) |
72
+ | `jupyter` | 0.07% | [Jupyter notebooks](https://huggingface.co/datasets/bigcode/jupyter-parsed) |
73
+ | `conala` | 0.07% | [CoNaLa](https://huggingface.co/datasets/neulab/conala) |
74
+
75
+ > _Realized mix may differ from target — supply-bound sources (pes2o, jupyter at this scale) route their deficit to FineWeb_.
76
 
77
  **Fine-tuning**
78
 
 
91
 
92
  | Benchmark | Few-shot | Metric | Score |
93
  |---|---|---|---|
94
+ | HellaSwag | 10-shot | acc_norm | 0.3098 |
95
+ | ARC-Easy | 25-shot | acc_norm | 0.4848 |
96
+ | ARC-Challenge | 25-shot | acc_norm | 0.2602 |
97
+ | MMLU | 5-shot | acc | 0.2512 |
98
+ | TruthfulQA | 0-shot | acc | 0.4268 |
99
+ | HumanEval | 0-shot | pass@1 | 0.0976 |
100
 
101
 
102
  ## Usage
code_completion_training_metadata.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": "finetune/configs/code_completion_125m.yaml",
3
+ "base_model": "results/runs/125m/sft_code/final",
4
+ "train_path": "/data/slm/data/runs/125m/code_completion/train.jsonl",
5
+ "val_path": "/data/slm/data/runs/125m/code_completion/val.jsonl",
6
+ "updates": 500,
7
+ "max_length": 768,
8
+ "micro_batch_size": 8,
9
+ "gradient_accumulation_steps": 4,
10
+ "learning_rate": 1e-05,
11
+ "weight_decay": 0.01,
12
+ "warmup_ratio": 0.05,
13
+ "best_val_loss": 0.8386935763888889
14
+ }
config.json CHANGED
@@ -8,7 +8,7 @@
8
  "AutoModelForCausalLM": "slm_remote.model.SLMForCausalLM"
9
  },
10
  "bos_token_id": 2,
11
- "dtype": "float32",
12
  "eos_token_id": 3,
13
  "hidden_size": 768,
14
  "initializer_range": 0.02,
 
8
  "AutoModelForCausalLM": "slm_remote.model.SLMForCausalLM"
9
  },
10
  "bos_token_id": 2,
11
+ "dtype": "bfloat16",
12
  "eos_token_id": 3,
13
  "hidden_size": 768,
14
  "initializer_range": 0.02,
generation_config.json CHANGED
@@ -1,12 +1,6 @@
1
  {
2
- "_from_model_config": true,
3
  "bos_token_id": 2,
4
- "eos_token_id": [
5
- 3
6
- ],
7
- "output_attentions": false,
8
- "output_hidden_states": false,
9
  "pad_token_id": 0,
10
- "transformers_version": "5.5.4",
11
- "use_cache": false
12
  }
 
1
  {
 
2
  "bos_token_id": 2,
3
+ "eos_token_id": 3,
 
 
 
 
4
  "pad_token_id": 0,
5
+ "do_sample": false
 
6
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:518b60bb67264ed34927f95d2bbf0ecd37d8ff71d3d712bb4b88073650314069
3
- size 501074816
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d597dff70f3c4946a168c19fd13cb272c1894e857e265501d675eb9af9c14881
3
+ size 250545608
tokenizer/tokenizer_config.json CHANGED
@@ -16,6 +16,7 @@
16
  "<|context|>",
17
  "<|endofcontext|>"
18
  ],
 
19
  "model_max_length": 1000000000000000019884624838656,
20
  "pad_token": "<PAD>",
21
  "tokenizer_class": "TokenizersBackend",
 
16
  "<|context|>",
17
  "<|endofcontext|>"
18
  ],
19
+ "is_local": true,
20
  "model_max_length": 1000000000000000019884624838656,
21
  "pad_token": "<PAD>",
22
  "tokenizer_class": "TokenizersBackend",