Text Generation
MLX
English
sol_milkshake
causal-lm
decoder-only
small-language-model
recurrent-depth
ngpt
research
Instructions to use solintellegence/Sol-Milkshake with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use solintellegence/Sol-Milkshake with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # if on a CUDA device, also pip install mlx[cuda] # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("solintellegence/Sol-Milkshake") prompt = "Once upon a time in" text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use solintellegence/Sol-Milkshake with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Generate some text mlx_lm.generate --model "solintellegence/Sol-Milkshake" --prompt "Once upon a time"
- Atomic Chat
Rewrite model card in plain language
Browse files
README.md
CHANGED
|
@@ -17,11 +17,11 @@ tags:
|
|
| 17 |
|
| 18 |
# Sol Milkshake
|
| 19 |
|
| 20 |
-
Sol Milkshake is a
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
##
|
| 25 |
|
| 26 |
| Setting | Value |
|
| 27 |
|---|---|
|
|
@@ -42,15 +42,15 @@ It uses hyperspherical optimization, shared transformer blocks, and local and ro
|
|
| 42 |
| Token embeddings | Tied to the output head |
|
| 43 |
| Weights | MLX NPZ |
|
| 44 |
|
| 45 |
-
##
|
| 46 |
|
| 47 |
-
Install MLX
|
| 48 |
|
| 49 |
```bash
|
| 50 |
pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
|
| 51 |
```
|
| 52 |
|
| 53 |
-
Download the repository
|
| 54 |
|
| 55 |
```python
|
| 56 |
from huggingface_hub import snapshot_download
|
|
@@ -73,7 +73,7 @@ text = generate(
|
|
| 73 |
print(text)
|
| 74 |
```
|
| 75 |
|
| 76 |
-
##
|
| 77 |
|
| 78 |
| Benchmark | Examples | Normalized accuracy |
|
| 79 |
|---|---:|---:|
|
|
@@ -83,22 +83,20 @@ print(text)
|
|
| 83 |
| PIQA | 1,838 | 52.50% |
|
| 84 |
| ArithMark-3 | 1,000 | 30.60% |
|
| 85 |
|
| 86 |
-
The reported Intelligence Index is 3.158. The four language-model tasks use full zero-shot splits with `lm-eval` 0.4.12. ArithMark-3 uses independent tokenization and normalized accuracy.
|
| 87 |
|
| 88 |
-
|
| 89 |
|
| 90 |
## Training
|
| 91 |
|
| 92 |
-
|
| 93 |
|
| 94 |
-
The tokenizer and prepared streams
|
| 95 |
|
| 96 |
-
## Files
|
| 97 |
|
| 98 |
-
`model.npz`
|
| 99 |
|
| 100 |
-
|
| 101 |
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
The model is licensed under [CC BY 4.0](LICENSE). Dataset licenses remain separate.
|
|
|
|
| 17 |
|
| 18 |
# Sol Milkshake
|
| 19 |
|
| 20 |
+
Sol Milkshake is a recurrent language model that runs with MLX on Apple Silicon. It has 2,990,000 parameters, and the released checkpoint received 2,526,565,888 token exposures across pretraining and recovery training.
|
| 21 |
|
| 22 |
+
The experiment combines hyperspherical optimization with shared transformer blocks. It also has two kinds of memory: local token-pattern memory and rolling memory across chunks. Milkshake completes text; it hasn't been instruction-tuned.
|
| 23 |
|
| 24 |
+
## What's in the model
|
| 25 |
|
| 26 |
| Setting | Value |
|
| 27 |
|---|---|
|
|
|
|
| 42 |
| Token embeddings | Tied to the output head |
|
| 43 |
| Weights | MLX NPZ |
|
| 44 |
|
| 45 |
+
## Try it on Apple Silicon
|
| 46 |
|
| 47 |
+
Install MLX and the tokenizer library, along with the Hub client:
|
| 48 |
|
| 49 |
```bash
|
| 50 |
pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
|
| 51 |
```
|
| 52 |
|
| 53 |
+
Download the repository, then import the model code and loader included here:
|
| 54 |
|
| 55 |
```python
|
| 56 |
from huggingface_hub import snapshot_download
|
|
|
|
| 73 |
print(text)
|
| 74 |
```
|
| 75 |
|
| 76 |
+
## Results and checkpoint selection
|
| 77 |
|
| 78 |
| Benchmark | Examples | Normalized accuracy |
|
| 79 |
|---|---:|---:|
|
|
|
|
| 83 |
| PIQA | 1,838 | 52.50% |
|
| 84 |
| ArithMark-3 | 1,000 | 30.60% |
|
| 85 |
|
| 86 |
+
The reported Intelligence Index is 3.158. The four language-model tasks use the full zero-shot splits with `lm-eval` 0.4.12. ArithMark-3 uses independent tokenization and normalized accuracy. The raw outputs are in `evals/`.
|
| 87 |
|
| 88 |
+
Recovery runs used this same suite to choose the checkpoint. That means the scores include the effect of checkpoint selection; they aren't an untouched held-out estimate. No independent evaluation has verified them.
|
| 89 |
|
| 90 |
## Training
|
| 91 |
|
| 92 |
+
Pretraining used FineWeb-Edu and FinePDFs-Edu, English UltraFineWeb multi-domain and question-answer subsets, Cosmopedia, and FineMath. Recovery mixed 70% of the original frozen curriculum with 15% Cosmopedia-v2 English and 15% FinePhrase.
|
| 93 |
|
| 94 |
+
The tokenizer and prepared streams stayed fixed during recovery, as did the existing filtering, deduplication, and decontamination rules. `training_state.json` records the run details.
|
| 95 |
|
| 96 |
+
## Files and limits
|
| 97 |
|
| 98 |
+
`model.npz` contains the MLX weights. The loader and generator are in `modeling_sol_milkshake.py`; `sol_config.py` holds the architecture settings. You'll also find `config.json`, tokenizer files, training metadata, and evaluation outputs here.
|
| 99 |
|
| 100 |
+
Milkshake is a small research model for studying recurrence, memory, and hyperspherical optimization, as well as MLX inference. It can produce repetitive or incoherent text and get answers wrong.
|
| 101 |
|
| 102 |
+
The model uses [CC BY 4.0](LICENSE). The training datasets keep their own licenses.
|
|
|
|
|
|