README.md

---
license: cc-by-nc-4.0
language:
  - en
library_name: pytorch
tags:
  - text-generation
  - instruction-tuning
  - from-scratch
  - educational
pipeline_tag: text-generation
---

# MiniMind-Instruct-137.7M

A ~137.7M-parameter GPT-style decoder transformer **built entirely from scratch** -
custom architecture, custom tokenizer, from-scratch pretraining, and instruction
fine-tuning. This is the **instruction-tuned** checkpoint: it follows the Alpaca prompt
format and responds to instructions.

> **Educational toy model.** Trained on a small subset for a book project. It is not
> intended to be useful or factual. See [Limitations](#limitations).

## Model details

| | |
|---|---|
| Architecture | GPT-style decoder (pre-norm, GELU FFN, absolute positional embeddings) |
| Parameters | 137.7M |
| Layers / Heads / d_model | 12 / 12 / 768 |
| Vocabulary | 32,000 (custom BPE, bundled) |
| Context length | 512 tokens |
| Checkpoint | best validation loss, step 1000 |

## Training

**Stage 1 - Pretraining (from scratch).** Next-token prediction on an OpenWebText subset,
~100k steps. Base checkpoint: `best_model.pt`.

**Stage 2 - Instruction fine-tuning (SFT).** [Alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca)
(52k) + [Dolly 2.0](https://huggingface.co/datasets/databricks/databricks-dolly-15k) (15k) =
~67k examples. Loss is masked to **response tokens only**; learning rate 5e-5 (10x lower
than pretraining).

| | Pretraining | SFT |
|---|---|---|
| Objective | Next-token | Response tokens only (prompt masked with -100) |
| Learning rate | 3e-4 | 5e-5 |
| Steps | ~100,000 | ~1,000-4,500 (early-stopped) |

Best validation loss: **see training curves**.

![Instruction fine-tuning loss curves](instruct_training_curves.png)

## How to use

This is a **custom `torch.nn.Module`**, not a HuggingFace `PreTrainedModel`, so it is **not**
`AutoModelForCausalLM`-loadable. Load it via the bundled [`model.py`](model.py):

```python
# pip install torch transformers
import torch
from transformers import AutoTokenizer
from model import load_model

REPO = "achavan1211/minimind-instruct-135m"
model, cfg = load_model("config.json", "model.pt", device="cpu")   # files from this repo
tok = AutoTokenizer.from_pretrained(REPO)

PROMPT = ("Below is an instruction that describes a task. "
          "Write a response that appropriately completes the request.\n\n"
          "### Instruction:\n{instruction}\n\n### Response:\n")

def ask(instruction, max_new_tokens=150):
    p = PROMPT.format(instruction=instruction)
    ids = tok.encode(p, return_tensors="pt")
    out = model.generate(ids, max_new_tokens=max_new_tokens, temperature=0.7,
                         top_p=0.9, repetition_penalty=1.2,
                         eos_id=tok.eos_token_id, stop_on_eos=True)
    return tok.decode(out[0], skip_special_tokens=True)[len(p):].strip()

print(ask("What is the capital of France?"))
# -> the capital of france is paris.
```

## Example outputs

**Q: Write a short poem about the ocean.**
> the sea was filled with life, like a river of stars and clouds;
> but the water had no shore, a sight so gentle and peaceful ...

**Q: Give three tips for staying healthy.**
> 1. get enough sleep to keep your body functioning properly.
> 2. be aware of your physical activity and take breaks throughout the day.
> 3. get plenty of sleep and focus on your health.

## Limitations

- **Low factual accuracy** - the model lacks capacity for reliable world knowledge; it will
  hallucinate confidently.
- **No multi-step reasoning** - it cannot do complex logic at ~137.7M parameters.
- **Best at format tasks** - short Q&A, reformatting, light creative writing within its
  training distribution.
- **Do not use for anything that matters.**

## Training data & license

| Data | Use | License / terms |
|---|---|---|
| OpenWebText (subset) | Pretraining | open |
| Alpaca (Stanford, 52k) | SFT | intended for **research use** |
| Dolly 2.0 (Databricks, 15k) | SFT | Apache-2.0 |

Because fine-tuning used Alpaca (research-use), this model is released under
**CC-BY-NC-4.0** (non-commercial). See [LICENSE](LICENSE).