Add a checkpoint list and a model-details spec

README.md

@@ -21,11 +21,36 @@ datasets:
 
 A 16,085,568-parameter recurrent decoder with post-decay pretraining weights. [Weevil-Recurrent-16M-QA](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-QA) is this model fine-tuned for multiple-choice questions.
 
-Pretraining used 9.75 billion tokens of the first mix, then the entire 750 million-token decay stage. Stored weights are float32.
+## Checkpoints
+
+| Model | Description |
+|---|---|
+| [ELBennett/Weevil-Recurrent-16M-Base](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-Base) | Post-decay pretraining checkpoint. |
+| [ELBennett/Weevil-Recurrent-16M-QA](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-QA) | The Base checkpoint fine-tuned for multiple-choice questions. |
+
+## Model details
+
+| Property | Value |
+|---|---|
+| Parameters | 16,085,568 |
+| Architecture | Causal Gated DeltaNet-2 |
+| Vocabulary | 50,368 |
+| Factorized embedding | 128 |
+| Hidden size | 256 |
+| Stack | 2-layer prelude + 4-layer core + 2-layer coda |
+| Pretraining | 9.75B first-mix tokens + the entire 750M decay stage = 10.5B content tokens |
+| Precision | Float32 stored weights; bfloat16 linears |
+| Pause token | `[pause]`, token id 50285 |
+
+## How to use
+
+```bash
+pip install torch transformers
+```
 
-## Use
+### Generate
 
-### Score choices
+Pass unpadded rows: right-padding is consumed by the convolution.
 
 ```python
 import torch
@@ -37,6 +62,17 @@ model = AutoModelForCausalLM.from_pretrained(
 tokenizer = AutoTokenizer.from_pretrained("ELBennett/Weevil-Recurrent-16M-Base")
 model.eval()
 
+prompt = "The capital of France is"
+prompt_ids = torch.tensor([tokenizer.encode(prompt, add_special_tokens=False)])
+output_ids = model.generate(prompt_ids, max_new_tokens=16)
+print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
+```
+
+### Score choices
+
+Continue with the model and tokenizer above.
+
+```python
 choices = ["cat", "dog"]
 prompt = "Which animal barks?\n\nA. cat\nB. dog\nAnswer:"
 pause_id = tokenizer.convert_tokens_to_ids("[pause]")
@@ -53,18 +89,9 @@ print(choices[int(scores.argmax())], [round(score, 3) for score in scores.tolist
 
 The prompt and pauses are prefilled once. Each choice is scored from its own copy of the recurrent state.
 
-### Generate
-
-Continue with the model, tokenizer, and prompt above. Pass unpadded rows: right-padding is consumed by the convolution.
-
-```python
-prompt_ids = torch.tensor([tokenizer.encode(prompt, add_special_tokens=False)])
-print(model.generate(prompt_ids, max_new_tokens=16).shape)
-```
-
 ### Reuse the recurrent state
 
-`generate` prefills the GDN2 state and convolution windows, then advances them one token at a time. `past_key_values` and `recurrent_state` refer to the same object: one state matrix and one convolution window per mixer, with a separate copy for every core pass.
+This continues the `prompt_ids` from the scoring example. `generate` prefills the GDN2 state and convolution windows, then advances them one token at a time. `past_key_values` and `recurrent_state` refer to the same object: one state matrix and one convolution window per mixer, with a separate copy for every core pass.
 
 ```python
 out = model(prompt_ids, use_cache=True)
@@ -116,13 +143,6 @@ Together that is **10.5 billion** content tokens: 9.75 billion from the first mi
 
 ## Architecture
 
-The model is a causal Gated DeltaNet-2 decoder with the same recurrence as the QA fine-tune.
-
-- Vocabulary: 50,368.
-- Factorized embedding: 128.
-- Hidden size: 256.
-- Stack: 2-layer prelude, 4-layer core, 2-layer coda.
-
 `core_passes` repeats only the core. Each core pass adds `input_scale * prelude_output` back onto the residual. Each mixer and MLP branch has a 1-lane read/write gate. Next-token logits go back through the factorized embedding. Linears run in bfloat16.
 
 `[pause]` is token id 50285. It is a special token, so the word "pause" in ordinary text is not rewritten to it. The model does not learn to emit pauses. Put them in the input.