|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- tokyotech-llm/swallow-code-v2 |
|
|
base_model: |
|
|
- meta-llama/Llama-3.2-1B |
|
|
- dudeperf3ct/codellm-tokenizer |
|
|
--- |
|
|
|
|
|
Writeup: https://dudeperf3ct.github.io/projects/train_llm_part2/ |
|
|
|
|
|
Repo: https://github.com/dudeperf3ct/minicode-llm/tree/main/codellm_pretrain/torch_titan |
|
|
|
|
|
This contains checkpoints every 5k steps for pretraining run 9.8B tokens using |
|
|
|
|
|
- Custom tokenizer: https://dudeperf3ct.github.io/projects/train_llm_part1/ |
|
|
- Dataset: [`tokyotech-llm/swallow-code-v2`](https://huggingface.co/datasets/tokyotech-llm/swallow-code-v2) |
|
|
- Model Architecture: Llama 3.2 1B (1 billion parameter) |
|
|
|
|
|
The repository contains detailed step on how to run evaluation using PyTorch DCP checkpoints. |
