| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: vllm |
| tags: |
| - causal-lm |
| - pytorch |
| - gqa |
| - research |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| --- |
| |
| # Dense-306 |
|
|
| Dense-306 is the 306.5M-parameter dense baseline from the matched run pair in |
| *Token Identity as a Routing Signal for Residual MLP Experts*. It is a |
| decoder-only base language model trained from scratch on approximately 8B |
| FineWeb-Edu tokens. |
|
|
| This is a pretrained base model, not an instruction-tuned or safety-aligned |
| assistant. |
|
|
| ## Architecture |
|
|
| - 18 transformer layers |
| - hidden size 1,024 |
| - 16 query heads and 4 key/value heads (GQA) |
| - QK-Norm and RoPE |
| - 2,048-token training context |
| - 32,000-token tokenizer |
| - dense SwiGLU width 4,096 |
| - 306,486,528 trainable parameters |
|
|
| ## Evaluation |
|
|
| Both models were evaluated zero-shot with EleutherAI LM Evaluation Harness |
| 0.4.12, identical seeds, BF16 inference, no sample limit, and a maximum context |
| of 2,048 tokens. |
|
|
| | Model | ARC-Easy acc_norm | PIQA acc_norm | HellaSwag acc_norm | WikiText-2 word PPL | |
| | --- | ---: | ---: | ---: | ---: | |
| | Dense-306 | 49.07 | 63.49 | 33.67 | 35.79 | |
| | TR-MOE-306 | 48.61 | 63.49 | 34.06 | 35.20 | |
| |
| The multiple-choice differences are smaller than one reported standard error. |
| These results support near-parity, not task-level superiority. |
| |
| At the last common point on a fixed FineWeb-Edu training-split evaluation |
| stream, the dense model reached NLL 2.9482 and the token-routed model reached |
| 2.9329. That stream is not an independent held-out set, and each architecture |
| was trained with one seed. |
| |
| ## Files and integrity |
| |
| `model.safetensors` is the exact FP32 model export used for the reported |
| downstream evaluation. |
| |
| - final training step: `7629` |
| - training checkpoint SHA-256: |
| `0d5bc1d8d8ead8ecde70e2705dfc8712e99a3de79bdf24f9b4e34fd1f668de7b` |
| - exported `model.safetensors` SHA-256: |
| `311c97b2233a0d182fe47fa641a427e30f59b49774872a85b59854457ed1e7ae` |
| |
| The repository also contains the tokenizer, generation configuration, and |
| `export_manifest.json`. |
|
|
| ## Runtime |
|
|
| The checkpoint uses the custom `DeepForCausalLM` format implemented by the |
| [Complexity-ML vLLM runtime](https://github.com/Complexity-ML/vllm-cuda_graph). |
| Stock Transformers and upstream vLLM do not currently register the `deep` |
| architecture. Use that runtime for inference, or adapt its loader before |
| loading these weights. |
|
|
| The tokenizer can be loaded independently: |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("Pacific-i64/Dense-306") |
| ``` |
|
|
| ## Limitations |
|
|
| This is a small research base model. It may generate inaccurate, biased, |
| offensive, repetitive, or unsafe text. It should not be used for high-stakes |
| decisions. The training corpus is web-derived and may contain errors, |
| copyrighted material, personal information, and social biases. |
|
|
| ## Paper and reproducibility |
|
|
| - [OpenReview submission](https://openreview.net/forum?id=Jd9jhTnkUy) |
| - [Anonymous supplementary code and results](https://github.com/Complexity-ML/tmlr-paper-pool) |
|
|
| The paper reports a matched single-seed observation. Multi-seed replication |
| remains open. |
|
|
| ## License |
|
|
| The Dense-306 model weights are released under the Apache License 2.0. See |
| `LICENSE`. |
|
|
