| --- |
| license: mit |
| language: |
| - en |
| datasets: |
| - roneneldan/TinyStories |
| - roneneldan/TinyStoriesInstruct |
| pipeline_tag: text-generation |
| tags: |
| - pytorch |
| - safetensors |
| - custom-code |
| --- |
| |
| # Sprout story model |
|
|
| A decoder-only model trained from scratch on TinyStoriesV2-GPT4 and instruction tuned on TinyStoriesInstruct. It has 29,368,832 parameters, 8 layers, width 512, 8 attention heads, a 512-token context, rope positions, RMSNorm, SwiGLU, tied embeddings, and a 4096-token byte-level BPE vocabulary. |
|
|
| This bundle contains float32 inference weights at instruction step 18,457. It excludes optimizer state, training data, credentials, and training dependencies. The tied embedding/output tensor is stored once and restored with `safetensors.torch.load_model`. `config.json` records weight and tokenizer SHA-256 checksums, source checkpoint SHA-256, source code commit, and the original training fingerprint. |
|
|
| [Try the live browser demo](https://huggingface.co/spaces/GeorgeCodesSometimes/sprout). Float32 ONNX inference runs on your device; prompts stay local. |
|
|
| ## Run locally |
|
|
| Download all files from this model repository into an empty directory, then run: |
|
|
| ```bash |
| python -m pip install -r requirements.txt |
| python app.py |
| ``` |
|
|
| Use the included loading module directly: |
|
|
| ```python |
| from inference import load, stream |
| |
| model, tok, config = load(".") |
| for text in stream(model, tok, {"Summary": "A little fox learns to be patient."}): |
| print(text, end="\r", flush=True) |
| ``` |
|
|
| `load` also accepts a Hub model ID and an optional `revision` commit. The included code is the loading contract; this is not a Transformers `AutoModel` repository. Loading does not require any training corpus files. The demo uses the exact training template, keeps the full instruction in context, and supports streamed output, stopping, temperature, and top-k. |
|
|
| ## Training and evaluation |
|
|
| The source checkpoint's instruction validation loss is 0.956147. The [training run](https://wandb.ai/george-nasseem/sprout/runs/rb9ndxvw) records configuration and losses. See the [project README](https://github.com/gnasseem/sprout#instruction-tuning) for the fixed behavioral evaluation, comparison with the pretrained model, and base-validation regression. Low validation loss does not establish reliable constraint following. |
|
|
| Pretraining uses the story-level seeded split in the project, with 537,686,596 training tokens and 5,402,089 validation tokens. Instruction preparation excludes malformed and duplicate stories, held-out story overlap, and conversations beyond the context limit. TinyStoriesInstruct is pinned to `ee050ed1f8720795be342921335e821856a2b42e`. These are synthetic English stories generated by GPT models, not a general conversational corpus. |
|
|
| ## Limitations and intended use |
|
|
| An educational story-generation experiment. It often misses requested words and sentences, repeats itself, and loses narrative details. Instruction tuning regresses base-story validation loss. Long instructions leave less room for the story. It has no general chat or factual-answering capability, no content filter, and no suitability evaluation for children. Outputs can contain biases or disturbing story content from its synthetic source corpus. |
|
|
| ## License and provenance |
|
|
| Code and exported weights use the included MIT license. The upstream [TinyStories dataset card](https://huggingface.co/datasets/roneneldan/TinyStories) lists CDLA-Sharing-1.0; [TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct) is the separate instruction corpus. No source stories are redistributed in this bundle. The dataset creators' [paper](https://arxiv.org/abs/2305.07759) describes TinyStories. |
|
|
| ## Published behavioral results |
|
|
| On 20 fixed held-out instruction prompts using matching CUDA bf16 settings, the tuned model included all requested words in 6/20 cases, included the exact requested sentence in 4/20, and emitted the assistant end token in 19/20. Base-story validation loss rose 11.81% after instruction tuning. This is a completed educational experiment with limited instruction following. [Complete unedited outputs](https://huggingface.co/GeorgeCodesSometimes/sprout-instruct/tree/main/evaluation) and the [project results](https://github.com/gnasseem/sprout#results) include settings and limitations. |
|
|
