---
library_name: transformers
tags:
- muse-spark
- testing
- random-weights
---

# muse-spark-tiny

**Randomly initialised** tiny Muse Spark checkpoint for shape/plumbing/TP smoke tests.
**No weight comes from the released checkpoint** — do not use for any quality evaluation.

Built from the release's `config.json` with exactly two knobs shrunk:

| | release | tiny |
|---|---|---|
| `text_config.num_hidden_layers` | 62 | **6** |
| `text_config.num_local_experts` | 256 | **8** |
| parameters | 822 B | **9.48 B** (1.15 %) |
| size (bf16) | 1644 GB | **18.97 GB**, 5 shards |

Everything else matches the release: vision tower 50 layers, audio encoder 4 stages,
`hidden_size` 8192, `moe_intermediate_size` 4096, vocab 202048 with an untied head,
plus the release tokenizer, processor config and chat template.

The per-layer lists `layer_types`, `layer_rope_theta` and `mlp_layer_types` were dropped
from the source config so `MuseSparkTextConfig.__post_init__` re-derives them at depth 6
rather than truncating the 62-entry lists — the `[w, w, w, 0]` sliding/NoPE pattern is
counted backward from the last layer, so NoPE correctly lands on layers 1 and 5.

Parameter split:

| component | params |
|---|---|
| text MoE blocks | 2.819 B |
| vision tower + projector | 1.928 B |
| embeddings | 1.655 B |
| lm_head (untied) | 1.655 B |
| text attention + norms | 1.309 B |
| audio encoder | 0.118 B |

Generated by `make_tiny_checkpoint.py` with `torch.manual_seed(0)`.