j0no12 commited on
Commit
2b6ad91
·
verified ·
1 Parent(s): c5c0500

Rewrite model card in plain language

Browse files
Files changed (1) hide show
  1. README.md +15 -17
README.md CHANGED
@@ -17,11 +17,11 @@ tags:
17
 
18
  # Sol Milkshake
19
 
20
- Sol Milkshake is a 2,990,000-parameter recurrent language model for MLX on Apple Silicon. Its checkpoint received 2,526,565,888 token exposures across pretraining and recovery training.
21
 
22
- It uses hyperspherical optimization, shared transformer blocks, and local and rolling memory. This is a base model for text completion.
23
 
24
- ## Model
25
 
26
  | Setting | Value |
27
  |---|---|
@@ -42,15 +42,15 @@ It uses hyperspherical optimization, shared transformer blocks, and local and ro
42
  | Token embeddings | Tied to the output head |
43
  | Weights | MLX NPZ |
44
 
45
- ## Use
46
 
47
- Install MLX, the tokenizer library, and the Hub client:
48
 
49
  ```bash
50
  pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
51
  ```
52
 
53
- Download the repository and use its included loader:
54
 
55
  ```python
56
  from huggingface_hub import snapshot_download
@@ -73,7 +73,7 @@ text = generate(
73
  print(text)
74
  ```
75
 
76
- ## Evaluation
77
 
78
  | Benchmark | Examples | Normalized accuracy |
79
  |---|---:|---:|
@@ -83,22 +83,20 @@ print(text)
83
  | PIQA | 1,838 | 52.50% |
84
  | ArithMark-3 | 1,000 | 30.60% |
85
 
86
- The reported Intelligence Index is 3.158. The four language-model tasks use full zero-shot splits with `lm-eval` 0.4.12. ArithMark-3 uses independent tokenization and normalized accuracy. Raw outputs are in `evals/`.
87
 
88
- The checkpoint was selected from recovery runs using this same evaluation suite. These scores therefore include checkpoint-selection effects and are not an untouched held-out estimate. They have not been independently verified.
89
 
90
  ## Training
91
 
92
- Initial pretraining used FineWeb-Edu, FinePDFs-Edu, English UltraFineWeb multi-domain and question-answer subsets, Cosmopedia, and FineMath. Recovery training mixed 70% of the original frozen curriculum with 15% Cosmopedia-v2 English and 15% FinePhrase.
93
 
94
- The tokenizer and prepared streams were fixed. Recovery retained the existing filtering, deduplication, and decontamination rules. Training details are recorded in `training_state.json`.
95
 
96
- ## Files
97
 
98
- `model.npz` holds the MLX weights. `modeling_sol_milkshake.py` includes the loader and generator, and `sol_config.py` contains the architecture settings. The repository also includes `config.json`, the tokenizer files, training metadata, and evaluation outputs.
99
 
100
- ## Limits and license
101
 
102
- Milkshake is a small research model for recurrence, memory, hyperspherical optimization, and MLX inference. It can produce repetitive or incoherent text and incorrect answers. It has not been instruction-tuned.
103
-
104
- The model is licensed under [CC BY 4.0](LICENSE). Dataset licenses remain separate.
 
17
 
18
  # Sol Milkshake
19
 
20
+ Sol Milkshake is a recurrent language model that runs with MLX on Apple Silicon. It has 2,990,000 parameters, and the released checkpoint received 2,526,565,888 token exposures across pretraining and recovery training.
21
 
22
+ The experiment combines hyperspherical optimization with shared transformer blocks. It also has two kinds of memory: local token-pattern memory and rolling memory across chunks. Milkshake completes text; it hasn't been instruction-tuned.
23
 
24
+ ## What's in the model
25
 
26
  | Setting | Value |
27
  |---|---|
 
42
  | Token embeddings | Tied to the output head |
43
  | Weights | MLX NPZ |
44
 
45
+ ## Try it on Apple Silicon
46
 
47
+ Install MLX and the tokenizer library, along with the Hub client:
48
 
49
  ```bash
50
  pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
51
  ```
52
 
53
+ Download the repository, then import the model code and loader included here:
54
 
55
  ```python
56
  from huggingface_hub import snapshot_download
 
73
  print(text)
74
  ```
75
 
76
+ ## Results and checkpoint selection
77
 
78
  | Benchmark | Examples | Normalized accuracy |
79
  |---|---:|---:|
 
83
  | PIQA | 1,838 | 52.50% |
84
  | ArithMark-3 | 1,000 | 30.60% |
85
 
86
+ The reported Intelligence Index is 3.158. The four language-model tasks use the full zero-shot splits with `lm-eval` 0.4.12. ArithMark-3 uses independent tokenization and normalized accuracy. The raw outputs are in `evals/`.
87
 
88
+ Recovery runs used this same suite to choose the checkpoint. That means the scores include the effect of checkpoint selection; they aren't an untouched held-out estimate. No independent evaluation has verified them.
89
 
90
  ## Training
91
 
92
+ Pretraining used FineWeb-Edu and FinePDFs-Edu, English UltraFineWeb multi-domain and question-answer subsets, Cosmopedia, and FineMath. Recovery mixed 70% of the original frozen curriculum with 15% Cosmopedia-v2 English and 15% FinePhrase.
93
 
94
+ The tokenizer and prepared streams stayed fixed during recovery, as did the existing filtering, deduplication, and decontamination rules. `training_state.json` records the run details.
95
 
96
+ ## Files and limits
97
 
98
+ `model.npz` contains the MLX weights. The loader and generator are in `modeling_sol_milkshake.py`; `sol_config.py` holds the architecture settings. You'll also find `config.json`, tokenizer files, training metadata, and evaluation outputs here.
99
 
100
+ Milkshake is a small research model for studying recurrence, memory, and hyperspherical optimization, as well as MLX inference. It can produce repetitive or incoherent text and get answers wrong.
101
 
102
+ The model uses [CC BY 4.0](LICENSE). The training datasets keep their own licenses.