File size: 4,297 Bytes
20a371d
 
 
 
 
 
 
 
 
 
 
 
 
 
559972f
20a371d
559972f
20a371d
559972f
20a371d
559972f
20a371d
559972f
70a25ae
559972f
70a25ae
 
20a371d
 
 
 
559972f
20a371d
 
 
 
 
559972f
20a371d
 
 
559972f
70a25ae
559972f
70a25ae
559972f
20a371d
559972f
20a371d
559972f
20a371d
559972f
20a371d
559972f
20a371d
559972f
20a371d
559972f
20a371d
559972f
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
---
license: mit
language:
- en
datasets:
- roneneldan/TinyStories
- roneneldan/TinyStoriesInstruct
pipeline_tag: text-generation
tags:
- pytorch
- safetensors
- custom-code
---

# Sprout story model

A decoder-only model trained from scratch on TinyStoriesV2-GPT4 and instruction tuned on TinyStoriesInstruct. It has 29,368,832 parameters, 8 layers, width 512, 8 attention heads, a 512-token context, rope positions, RMSNorm, SwiGLU, tied embeddings, and a 4096-token byte-level BPE vocabulary.

This bundle contains float32 inference weights at instruction step 18,457. It excludes optimizer state, training data, credentials, and training dependencies. The tied embedding/output tensor is stored once and restored with `safetensors.torch.load_model`. `config.json` records weight and tokenizer SHA-256 checksums, source checkpoint SHA-256, source code commit, and the original training fingerprint.

[Try the live browser demo](https://huggingface.co/spaces/GeorgeCodesSometimes/sprout). Float32 ONNX inference runs on your device; prompts stay local.

## Run locally

Download all files from this model repository into an empty directory, then run:

```bash
python -m pip install -r requirements.txt
python app.py
```

Use the included loading module directly:

```python
from inference import load, stream

model, tok, config = load(".")
for text in stream(model, tok, {"Summary": "A little fox learns to be patient."}):
    print(text, end="\r", flush=True)
```

`load` also accepts a Hub model ID and an optional `revision` commit. The included code is the loading contract; this is not a Transformers `AutoModel` repository. Loading does not require any training corpus files. The demo uses the exact training template, keeps the full instruction in context, and supports streamed output, stopping, temperature, and top-k.

## Training and evaluation

The source checkpoint's instruction validation loss is 0.956147. The [training run](https://wandb.ai/george-nasseem/sprout/runs/rb9ndxvw) records configuration and losses. See the [project README](https://github.com/gnasseem/sprout#instruction-tuning) for the fixed behavioral evaluation, comparison with the pretrained model, and base-validation regression. Low validation loss does not establish reliable constraint following.

Pretraining uses the story-level seeded split in the project, with 537,686,596 training tokens and 5,402,089 validation tokens. Instruction preparation excludes malformed and duplicate stories, held-out story overlap, and conversations beyond the context limit. TinyStoriesInstruct is pinned to `ee050ed1f8720795be342921335e821856a2b42e`. These are synthetic English stories generated by GPT models, not a general conversational corpus.

## Limitations and intended use

An educational story-generation experiment. It often misses requested words and sentences, repeats itself, and loses narrative details. Instruction tuning regresses base-story validation loss. Long instructions leave less room for the story. It has no general chat or factual-answering capability, no content filter, and no suitability evaluation for children. Outputs can contain biases or disturbing story content from its synthetic source corpus.

## License and provenance

Code and exported weights use the included MIT license. The upstream [TinyStories dataset card](https://huggingface.co/datasets/roneneldan/TinyStories) lists CDLA-Sharing-1.0; [TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct) is the separate instruction corpus. No source stories are redistributed in this bundle. The dataset creators' [paper](https://arxiv.org/abs/2305.07759) describes TinyStories.

## Published behavioral results

On 20 fixed held-out instruction prompts using matching CUDA bf16 settings, the tuned model included all requested words in 6/20 cases, included the exact requested sentence in 4/20, and emitted the assistant end token in 19/20. Base-story validation loss rose 11.81% after instruction tuning. This is a completed educational experiment with limited instruction following. [Complete unedited outputs](https://huggingface.co/GeorgeCodesSometimes/sprout-instruct/tree/main/evaluation) and the [project results](https://github.com/gnasseem/sprout#results) include settings and limitations.