HyperCloned from facebook/MobileLLM-R1-140M-base (emb×2, ffn×2)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,43 @@
+---
+license: other
+base_model: facebook/MobileLLM-R1-140M-base
+tags:
+  - hypercloning
+  - model-growth
+  - initialization
+  - mobilellm
+---
+
+# MobileLLM-R1-140M-base — 2× HyperCloned
+
+Initialized from [facebook/MobileLLM-R1-140M-base](https://huggingface.co/facebook/MobileLLM-R1-140M-base)
+using **HyperCloning** ([Samragh et al., 2024](https://arxiv.org/abs/2409.12903)).
+
+## Architecture
+
+| | Source | HyperCloned |
+|---|---|---|
+| hidden_size | 576 | 1152 |
+| num_attention_heads | 9 | 18 |
+| num_key_value_heads | 3 | 6 |
+| head_dim | 64 | 64 |
+| intermediate_size | 8192 | 16384 |
+| num_layers | 15 | 15 |
+| parameters | 140,248,512 | 454,790,016 |
+
+## Method
+
+Each weight W is expanded via `W.repeat(n, n) / n` (the paper's W/2 for 2×).
+Heads double with embedding dimension. head_dim is preserved.
+Output logits match the source model at initialization.
+
+This is an **initialization checkpoint** — further training is needed.
+
+```bibtex
+@article{samragh2024hypercloning,
+  title={Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization},
+  author={Samragh et al.},
+  journal={arXiv:2409.12903},
+  year={2024}
+}
+```

config.json

@@ -0,0 +1,27 @@
+{
+  "transformers_version": "5.5.3",
+  "architectures": [
+    "Llama4ForCausalLM"
+  ],
+  "vocab_size": 128256,
+  "hidden_size": 1152,
+  "intermediate_size": 16384,
+  "num_hidden_layers": 15,
+  "num_attention_heads": 18,
+  "num_key_value_heads": 6,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "max_position_embeddings": 4096,
+  "rms_norm_eps": 1e-05,
+  "use_cache": true,
+  "bos_token_id": 128000,
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "tie_word_embeddings": false,
+  "attention_dropout": 0.0,
+  "attention_bias": false,
+  "model_type": "llama4_text"
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 128000,
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.5.3",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1283db32d83f28624b58a3508cde70754a11031185091f3829ee2e57fb3a1308
+size 1819176256

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 4096,
+  "padding_side": "right",
+  "tokenizer_class": "TokenizersBackend"
+}