Upload folder using huggingface_hub

README.md

@@ -1,3 +1,5 @@
-Nano GPT Model
+# KrestGPT
+
+Обученая модель NanoGPT из примера Andrej Karpathy: https://github.com/karpathy/nanochat/tree/master
 
 I did it for lulz

report/base-model-evaluation.md

@@ -0,0 +1,28 @@
+## Base model evaluation
+timestamp: 2025-10-15 11:43:07
+
+- Model: base_model (step 21400)
+- CORE metric: 0.2069
+- hellaswag_zeroshot: 0.2584
+- jeopardy: 0.0619
+- bigbench_qa_wikidata: 0.5099
+- arc_easy: 0.5146
+- arc_challenge: 0.1251
+- copa: 0.2800
+- commonsense_qa: 0.2342
+- piqa: 0.3711
+- openbook_qa: 0.1227
+- lambada_openai: 0.3761
+- hellaswag: 0.2569
+- winograd: 0.2821
+- winogrande: 0.0718
+- bigbench_dyck_languages: 0.1070
+- agi_eval_lsat_ar: 0.0924
+- bigbench_cs_algorithms: 0.3765
+- bigbench_operators: 0.1524
+- bigbench_repeat_copy_logic: 0.0000
+- squad: 0.2328
+- coqa: 0.2036
+- boolq: -0.2587
+- bigbench_language_identification: 0.1816
+

report/base-model-loss.md

@@ -0,0 +1,14 @@
+## Base model loss
+timestamp: 2025-10-15 11:14:18
+
+- train bpb: 0.8171
+- val bpb: 0.8145
+- sample 0: <|bos|>The capital of France is Paris, which is located in the north of the country. The city is the
+- sample 1: <|bos|>The chemical symbol of gold is Au. The chemical symbol of gold is Au. The chemical symbol of gold is
+- sample 2: <|bos|>If yesterday was Friday, then tomorrow will be Friday. That’s because tomorrow is Friday, and Friday is Friday. That’s
+- sample 3: <|bos|>The opposite of hot is cold. The opposite of cold is hot. The opposite of hot is cold.
+- sample 4: <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
+- sample 5: <|bos|>My favorite color is red. I am a redhead. I am a redhead. I am
+- sample 6: <|bos|>If 5*x + 3 = 13, then x is 13.
+If 5*x + 3 = 13, then
+

report/base-model-training.md

@@ -0,0 +1,39 @@
+## Base model training
+timestamp: 2025-10-15 11:11:26
+
+- run: d26
+- depth: 20
+- max_seq_len: 2048
+- num_iterations: -1
+- target_flops: -1.0000
+- target_param_data_ratio: 20
+- device_batch_size: 32
+- total_batch_size: 524,288
+- embedding_lr: 0.2000
+- unembedding_lr: 0.0040
+- weight_decay: 0.0000
+- matrix_lr: 0.0200
+- grad_clip: 1.0000
+- eval_every: 250
+- eval_tokens: 10,485,760
+- core_metric_every: 2000
+- core_metric_max_per_task: 500
+- sample_every: 2000
+- model_tag: 
+- Number of parameters: 560,988,160
+- Number of FLOPs per token: 3.491758e+09
+- Calculated number of iterations: 21,400
+- Number of training tokens: 11,219,763,200
+- Tokens : Params ratio: 20.0000
+- DDP world size: 1
+- warmup_ratio: 0.0000
+- warmdown_ratio: 0.2000
+- final_lr_frac: 0.0000
+- Minimum validation bpb: 0.8143
+- Final validation bpb: 0.8143
+- CORE metric estimate: 0.2122
+- MFU %: 46.54%
+- Total training flops: 3.917670e+19
+- Total training time: 1331.57m
+- Peak memory usage: 79349.52MiB
+

report/chat-evaluation-mid.md

@@ -0,0 +1,21 @@
+## Chat evaluation mid
+timestamp: 2025-10-15 13:24:50
+
+- source: mid
+- task_name: None
+- dtype: bfloat16
+- temperature: 0.0000
+- max_new_tokens: 512
+- num_samples: 1
+- top_k: 50
+- batch_size: 8
+- model_tag: None
+- step: None
+- max_problems: None
+- ARC-Easy: 0.3906
+- ARC-Challenge: 0.2739
+- MMLU: 0.3094
+- GSM8K: 0.0273
+- HumanEval: 0.0671
+- ChatCORE metric: 0.0786
+

report/chat-evaluation-sft.md

@@ -0,0 +1,21 @@
+## Chat evaluation sft
+timestamp: 2025-10-15 14:40:31
+
+- source: sft
+- task_name: None
+- dtype: bfloat16
+- temperature: 0.0000
+- max_new_tokens: 512
+- num_samples: 1
+- top_k: 50
+- batch_size: 8
+- model_tag: None
+- step: None
+- max_problems: None
+- ARC-Easy: 0.4154
+- ARC-Challenge: 0.3157
+- MMLU: 0.3198
+- GSM8K: 0.0387
+- HumanEval: 0.0732
+- ChatCORE metric: 0.1026
+

report/chat-sft.md

@@ -0,0 +1,23 @@
+## Chat SFT
+timestamp: 2025-10-15 13:53:40
+
+- run: d26
+- source: mid
+- dtype: bfloat16
+- device_batch_size: 4
+- num_epochs: 1
+- max_iterations: -1
+- target_examples_per_step: 32
+- unembedding_lr: 0.0040
+- embedding_lr: 0.2000
+- matrix_lr: 0.0200
+- weight_decay: 0.0000
+- init_lr_frac: 0.0200
+- eval_every: 100
+- eval_steps: 100
+- eval_metrics_every: 200
+- Training rows: 20,843
+- Number of iterations: 651
+- Training loss: 1.0478
+- Validation loss: 1.0655
+

report/header.md

@@ -0,0 +1,36 @@
+# nanochat training report
+
+Generated: 2025-10-14 11:16:29
+
+## Environment
+
+### Git Information
+- Branch: master
+- Commit: dd6ff9a (dirty)
+- Message: fix bug in fallback case of find_largest_model
+
+### Hardware
+- Platform: Linux
+- CPUs: 20 cores (20 logical)
+- Memory: 235.9 GB
+- GPUs: 1x NVIDIA H100 80GB HBM3
+- GPU Memory: 79.2 GB total
+- CUDA Version: 12.8
+- Hourly Rate: $3.00/hour
+
+### Software
+- Python: 3.10.12
+- PyTorch: 2.8.0+cu128
+
+
+### Bloat
+- Characters: 39,122
+- Lines: 899
+- Files: 9
+- Tokens (approx): 9,780
+- Dependencies (uv.lock lines): 2,004
+
+Run started: 2025-10-14 11:16:29
+
+---
+

report/midtraining.md

@@ -0,0 +1,20 @@
+## Midtraining
+timestamp: 2025-10-15 12:37:55
+
+- run: d26
+- dtype: bfloat16
+- max_seq_len: 2048
+- device_batch_size: 32
+- unembedding_lr: 0.0040
+- embedding_lr: 0.2000
+- matrix_lr: 0.0200
+- init_lr_frac: 1.0000
+- weight_decay: 0.0000
+- final_lr_frac: 0.0000
+- eval_every: 150
+- eval_tokens: 10,485,760
+- total_batch_size: 524,288
+- Number of iterations: 771
+- DDP world size: 1
+- Minimum validation bpb: 0.4171
+

report/report.md

@@ -0,0 +1,270 @@
+# nanochat training report
+
+Generated: 2025-10-14 11:16:29
+
+## Environment
+
+### Git Information
+- Branch: master
+- Commit: dd6ff9a (dirty)
+- Message: fix bug in fallback case of find_largest_model
+
+### Hardware
+- Platform: Linux
+- CPUs: 20 cores (20 logical)
+- Memory: 235.9 GB
+- GPUs: 1x NVIDIA H100 80GB HBM3
+- GPU Memory: 79.2 GB total
+- CUDA Version: 12.8
+- Hourly Rate: $3.00/hour
+
+### Software
+- Python: 3.10.12
+- PyTorch: 2.8.0+cu128
+
+
+### Bloat
+- Characters: 39,122
+- Lines: 899
+- Files: 9
+- Tokens (approx): 9,780
+- Dependencies (uv.lock lines): 2,004
+
+Run started: 2025-10-14 11:16:29
+
+---
+
+## Tokenizer training
+timestamp: 2025-10-14 11:17:38
+
+- max_chars: 2,000,000,000
+- doc_cap: 10,000
+- vocab_size: 65,536
+- train_time: 65.5560
+- num_special_tokens: 9
+- token_bytes_min: 1
+- token_bytes_max: 32
+- token_bytes_mean: 6.9151
+- token_bytes_std: 2.8736
+
+
+## Tokenizer evaluation
+timestamp: 2025-10-14 11:17:42
+
+### Comparison with GPT-2
+
+| Text Type | Bytes | GPT-2 Tokens | GPT-2 Ratio | Ours Tokens | Ours Ratio | Relative Diff % |
+|-----------|-------|--------------|--------------|-------------|------------|-----------------|
+| news | 1819 | 404 | 4.50 | 375 | 4.85 | +7.2% |
+| korean | 893 | 745 | 1.20 | 721 | 1.24 | +3.2% |
+| code | 1259 | 576 | 2.19 | 493 | 2.55 | +14.4% |
+| math | 1834 | 936 | 1.96 | 966 | 1.90 | -3.2% |
+| science | 1112 | 260 | 4.28 | 225 | 4.94 | +13.5% |
+| fwe-train | 4208518 | 900364 | 4.67 | 856901 | 4.91 | +4.8% |
+| fwe-val | 4908443 | 1059062 | 4.63 | 1010356 | 4.86 | +4.6% |
+
+### Comparison with GPT-4
+
+| Text Type | Bytes | GPT-4 Tokens | GPT-4 Ratio | Ours Tokens | Ours Ratio | Relative Diff % |
+|-----------|-------|--------------|--------------|-------------|------------|-----------------|
+| news | 1819 | 387 | 4.70 | 375 | 4.85 | +3.1% |
+| korean | 893 | 364 | 2.45 | 721 | 1.24 | -98.1% |
+| code | 1259 | 309 | 4.07 | 493 | 2.55 | -59.5% |
+| math | 1834 | 832 | 2.20 | 966 | 1.90 | -16.1% |
+| science | 1112 | 249 | 4.47 | 225 | 4.94 | +9.6% |
+| fwe-train | 4208518 | 874799 | 4.81 | 856901 | 4.91 | +2.0% |
+| fwe-val | 4908443 | 1029691 | 4.77 | 1010356 | 4.86 | +1.9% |
+
+
+## Base model training
+timestamp: 2025-10-15 11:11:26
+
+- run: d26
+- depth: 20
+- max_seq_len: 2048
+- num_iterations: -1
+- target_flops: -1.0000
+- target_param_data_ratio: 20
+- device_batch_size: 32
+- total_batch_size: 524,288
+- embedding_lr: 0.2000
+- unembedding_lr: 0.0040
+- weight_decay: 0.0000
+- matrix_lr: 0.0200
+- grad_clip: 1.0000
+- eval_every: 250
+- eval_tokens: 10,485,760
+- core_metric_every: 2000
+- core_metric_max_per_task: 500
+- sample_every: 2000
+- model_tag: 
+- Number of parameters: 560,988,160
+- Number of FLOPs per token: 3.491758e+09
+- Calculated number of iterations: 21,400
+- Number of training tokens: 11,219,763,200
+- Tokens : Params ratio: 20.0000
+- DDP world size: 1
+- warmup_ratio: 0.0000
+- warmdown_ratio: 0.2000
+- final_lr_frac: 0.0000
+- Minimum validation bpb: 0.8143
+- Final validation bpb: 0.8143
+- CORE metric estimate: 0.2122
+- MFU %: 46.54%
+- Total training flops: 3.917670e+19
+- Total training time: 1331.57m
+- Peak memory usage: 79349.52MiB
+
+
+## Base model loss
+timestamp: 2025-10-15 11:14:18
+
+- train bpb: 0.8171
+- val bpb: 0.8145
+- sample 0: <|bos|>The capital of France is Paris, which is located in the north of the country. The city is the
+- sample 1: <|bos|>The chemical symbol of gold is Au. The chemical symbol of gold is Au. The chemical symbol of gold is
+- sample 2: <|bos|>If yesterday was Friday, then tomorrow will be Friday. That’s because tomorrow is Friday, and Friday is Friday. That’s
+- sample 3: <|bos|>The opposite of hot is cold. The opposite of cold is hot. The opposite of hot is cold.
+- sample 4: <|bos|>The planets of the solar system are: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune,
+- sample 5: <|bos|>My favorite color is red. I am a redhead. I am a redhead. I am
+- sample 6: <|bos|>If 5*x + 3 = 13, then x is 13.
+If 5*x + 3 = 13, then
+
+
+## Base model evaluation
+timestamp: 2025-10-15 11:43:07
+
+- Model: base_model (step 21400)
+- CORE metric: 0.2069
+- hellaswag_zeroshot: 0.2584
+- jeopardy: 0.0619
+- bigbench_qa_wikidata: 0.5099
+- arc_easy: 0.5146
+- arc_challenge: 0.1251
+- copa: 0.2800
+- commonsense_qa: 0.2342
+- piqa: 0.3711
+- openbook_qa: 0.1227
+- lambada_openai: 0.3761
+- hellaswag: 0.2569
+- winograd: 0.2821
+- winogrande: 0.0718
+- bigbench_dyck_languages: 0.1070
+- agi_eval_lsat_ar: 0.0924
+- bigbench_cs_algorithms: 0.3765
+- bigbench_operators: 0.1524
+- bigbench_repeat_copy_logic: 0.0000
+- squad: 0.2328
+- coqa: 0.2036
+- boolq: -0.2587
+- bigbench_language_identification: 0.1816
+
+
+## Midtraining
+timestamp: 2025-10-15 12:37:55
+
+- run: d26
+- dtype: bfloat16
+- max_seq_len: 2048
+- device_batch_size: 32
+- unembedding_lr: 0.0040
+- embedding_lr: 0.2000
+- matrix_lr: 0.0200
+- init_lr_frac: 1.0000
+- weight_decay: 0.0000
+- final_lr_frac: 0.0000
+- eval_every: 150
+- eval_tokens: 10,485,760
+- total_batch_size: 524,288
+- Number of iterations: 771
+- DDP world size: 1
+- Minimum validation bpb: 0.4171
+
+
+## Chat evaluation mid
+timestamp: 2025-10-15 13:24:50
+
+- source: mid
+- task_name: None
+- dtype: bfloat16
+- temperature: 0.0000
+- max_new_tokens: 512
+- num_samples: 1
+- top_k: 50
+- batch_size: 8
+- model_tag: None
+- step: None
+- max_problems: None
+- ARC-Easy: 0.3906
+- ARC-Challenge: 0.2739
+- MMLU: 0.3094
+- GSM8K: 0.0273
+- HumanEval: 0.0671
+- ChatCORE metric: 0.0786
+
+
+## Chat SFT
+timestamp: 2025-10-15 13:53:40
+
+- run: d26
+- source: mid
+- dtype: bfloat16
+- device_batch_size: 4
+- num_epochs: 1
+- max_iterations: -1
+- target_examples_per_step: 32
+- unembedding_lr: 0.0040
+- embedding_lr: 0.2000
+- matrix_lr: 0.0200
+- weight_decay: 0.0000
+- init_lr_frac: 0.0200
+- eval_every: 100
+- eval_steps: 100
+- eval_metrics_every: 200
+- Training rows: 20,843
+- Number of iterations: 651
+- Training loss: 1.0478
+- Validation loss: 1.0655
+
+
+## Chat evaluation sft
+timestamp: 2025-10-15 14:40:31
+
+- source: sft
+- task_name: None
+- dtype: bfloat16
+- temperature: 0.0000
+- max_new_tokens: 512
+- num_samples: 1
+- top_k: 50
+- batch_size: 8
+- model_tag: None
+- step: None
+- max_problems: None
+- ARC-Easy: 0.4154
+- ARC-Challenge: 0.3157
+- MMLU: 0.3198
+- GSM8K: 0.0387
+- HumanEval: 0.0732
+- ChatCORE metric: 0.1026
+
+
+## Summary
+
+- Characters: 39,122
+- Lines: 899
+- Files: 9
+- Tokens (approx): 9,780
+- Dependencies (uv.lock lines): 2,004
+
+| Metric          | BASE     | MID      | SFT      | RL       |
+|-----------------|----------|----------|----------|----------|
+| CORE            | 0.2069   | -        | -        | -        |
+| ARC-Challenge   | -        | 0.2739   | 0.3157   | -        |
+| ARC-Easy        | -        | 0.3906   | 0.4154   | -        |
+| GSM8K           | -        | 0.0273   | 0.0387   | -        |
+| HumanEval       | -        | 0.0671   | 0.0732   | -        |
+| MMLU            | -        | 0.3094   | 0.3198   | -        |
+| ChatCORE        | -        | 0.0786   | 0.1026   | -        |
+
+Total wall clock time: 27h24m

report/tokenizer-evaluation.md

@@ -0,0 +1,27 @@
+## Tokenizer evaluation
+timestamp: 2025-10-14 11:17:42
+
+### Comparison with GPT-2
+
+| Text Type | Bytes | GPT-2 Tokens | GPT-2 Ratio | Ours Tokens | Ours Ratio | Relative Diff % |
+|-----------|-------|--------------|--------------|-------------|------------|-----------------|
+| news | 1819 | 404 | 4.50 | 375 | 4.85 | +7.2% |
+| korean | 893 | 745 | 1.20 | 721 | 1.24 | +3.2% |
+| code | 1259 | 576 | 2.19 | 493 | 2.55 | +14.4% |
+| math | 1834 | 936 | 1.96 | 966 | 1.90 | -3.2% |
+| science | 1112 | 260 | 4.28 | 225 | 4.94 | +13.5% |
+| fwe-train | 4208518 | 900364 | 4.67 | 856901 | 4.91 | +4.8% |
+| fwe-val | 4908443 | 1059062 | 4.63 | 1010356 | 4.86 | +4.6% |
+
+### Comparison with GPT-4
+
+| Text Type | Bytes | GPT-4 Tokens | GPT-4 Ratio | Ours Tokens | Ours Ratio | Relative Diff % |
+|-----------|-------|--------------|--------------|-------------|------------|-----------------|
+| news | 1819 | 387 | 4.70 | 375 | 4.85 | +3.1% |
+| korean | 893 | 364 | 2.45 | 721 | 1.24 | -98.1% |
+| code | 1259 | 309 | 4.07 | 493 | 2.55 | -59.5% |
+| math | 1834 | 832 | 2.20 | 966 | 1.90 | -16.1% |
+| science | 1112 | 249 | 4.47 | 225 | 4.94 | +9.6% |
+| fwe-train | 4208518 | 874799 | 4.81 | 856901 | 4.91 | +2.0% |
+| fwe-val | 4908443 | 1029691 | 4.77 | 1010356 | 4.86 | +1.9% |
+

report/tokenizer-training.md

@@ -0,0 +1,13 @@
+## Tokenizer training
+timestamp: 2025-10-14 11:17:38
+
+- max_chars: 2,000,000,000
+- doc_cap: 10,000
+- vocab_size: 65,536
+- train_time: 65.5560
+- num_special_tokens: 9
+- token_bytes_min: 1
+- token_bytes_max: 32
+- token_bytes_mean: 6.9151
+- token_bytes_std: 2.8736
+