| --- |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| datasets: |
| - epfml/FineWeb-HQ |
| - HuggingFaceTB/smollm-corpus |
| tags: |
| - causal-lm |
| - base-model |
| - muon |
| - xsa-refresh |
| - custom-code |
| - trust-remote-code |
| --- |
| |
| # BananaMind 2 Pico Test - 90% |
|
|
| This is the **90% checkpoint** of a 900,002-parameter |
| base causal language model. It is not instruction tuned. |
|
|
| ## Architecture |
|
|
| | Field | Value | |
| |---|---:| |
| | Parameters | 900,002 | |
| | Layers / hidden size | 6 / 96 | |
| | SwiGLU intermediate size | 380 | |
| | Query / KV heads | 6 / 2 | |
| | Head dimension | 16 | |
| | Context | 4,096 | |
| | Vocabulary | 384, tied | |
| | Refresh layers | 4 and 6 | |
| | Refresh kernel | Causal depthwise, width 9 | |
|
|
| The selective XSA refresh gate reads detached attention output as its signal, |
| reinjects the original input embedding as its value, and carries convolution |
| history alongside the K/V cache. Its learned residual scalar starts at zero. |
|
|
| ## Training |
|
|
| | Field | Value | |
| |---|---:| |
| | Progress | 90% | |
| | Tokens seen | 180,004,847,616 | |
| | Target tokens | 200,000,000,000 | |
| | Hardware | 4 x NVIDIA H200 | |
| | Matrix optimizer | Stock `torch.optim.Muon` | |
| | Muon peak LR | 0.07 | |
| | Embedding/control optimizer | AdamW, LR 0.004 | |
| | Precision | bfloat16 autocast | |
|
|
| | Token range | FineWeb-HQ | Cosmopedia v2 | |
| |---|---:|---:| |
| | 0.00B-100.00B | 80% | 20% | |
| | 100.00B-200.00B | 60% | 40% | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "Banaxi-Tech/pico-test" |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True) |
| ``` |
|
|
