| --- |
| license: apache-2.0 |
| language: |
| - code |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - causal-lm |
| - code |
| - custom-code |
| - partially-looped |
| - n-gram |
| --- |
| |
| # BananaMind 2.1 MiniCoder |
|
|
| BananaMind 2.1 MiniCoder is an under-25M-parameter code language model. |
|
|
| - 24,949,999 total parameters |
| - 19,950,029 Transformer parameters |
| - 4,999,970 parameters in causal bigram/four-gram memory |
| - BananaMind 2 Nano's 8,192-token tokenizer |
| - 13 physical layers and 18 effective passes |
| - execution: `L1 → L2 → L3 → L4 → L5 → L6 → L7 → L8 → L9 → L5 → L6 → L7 → L8 → L9 → L10 → L11 → L12 → L13` |
| - the complete L5-L9 middle stack shares weights across its two passes |
| - 4,096-token context |
|
|
| The n-gram module has independent 51,699-entry bigram and four-gram hash |
| tables, each with 48-dimensional values. Their concatenated representation is |
| projected to the 384-wide residual stream. It is injected through separate |
| learned gates at the beginning of both middle-stack passes. |
|
|
| ## Training data |
|
|
| The exact 30B-token streamed mixture is: |
|
|
| | Source | Tokens | Share | |
| |---|---:|---:| |
| | [The Stack v3 train](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train) | 22.5B | 75% | |
| | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | 7.5B | 25% | |
|
|
| Stack v3 is streamed as repository-ordered source files. Vendored files are |
| skipped, while repository path, file path, and detected language are included |
| in the training text. FineWeb-Edu supplies prose, naming, comments, and general |
| language knowledge. |
|
|
| Checkpoints are uploaded every 5% with safetensors, tokenizer files, metrics, |
| pinned dataset revisions, exact source-token accounting, and optimizer state. |
|
|
| ## Launch |
|
|
| ```bash |
| ./launch_training_hf_job.sh 4 fresh |
| ./launch_training_hf_job.sh 4 resume |
| ./launch_training_hf_job.sh 8 resume |
| ``` |
|
|
|
|
