README.md

---
license: mit
language:
- en
tags:
- tiny
- slm
- tlm
- llm
- small
- question-generator
- harley-ml
- small-language-model
- experiment
- experimental
- text-generation
- question-generation
- questions
- question
---

# StopAskingQuestionsMini-656k
This model is small. Well, that's an understatement. But welcome to the world of tiny language models.
StopAskingQuestionsMini is a six-hundred and fifty-six thousand parameter language model trained on roughly 23 million tokens of questions without answers. That may sound counterintuitive:
> What is the point of generating questions with no answer?

There is no practical reason for doing so. However, this model wasn't built for practical use, it was built to answer the ongoing question that I am trying to answer:
> How much intellect can you stuff into a tiny model before it collapses?

This project, or any of our projects, don't truly answer this - because every day, there is always a new advancement. For example, DeepSeek created [Engram](https://arxiv.org/pdf/2601.07372), a novel architecture component that increases knowledge storage at very low compute cost.

Furthermore, 

> What can this model even do?

Not much. It can generate partially coherent questions, and that's pretty much it.

## Architecture

StopAskingQuestionsMini uses a scaled down version of the [Qwen3](https://arxiv.org/abs/2505.09388) architecture.


| Parameter | Value |
|-----------|-------|
| Hidden Layers | 2 |
| Hidden Size | 128 |
| Attention Heads | 2 |
| KV Heads | 2 |
| Intermediate Size | 512 |
| RoPE Theta | 10000.0 |
| Max Position Embeddings | 96 |
| Tie Word Embeddings | True |
| Vocab Size | 1024 |

## Training

StopAskingQuestionsMini trained on 23 million tokens of questions for two epochs with a batch size of 16.

### Training Results


| Epoch | Train Loss | Eval Loss | Train PPL | Eval PPL |
|-------|------------|-----------|-----------|----------|
| 0.07  | 4.0797     | 3.0011    | 59.05     | 20.11    |
| 0.22  | 2.6331     | 2.5703    | 13.92     | 13.07    |
| 0.37  | 2.4906     | 2.4586    | 12.07     | 11.68    |
| 0.52  | 2.4213     | 2.3989    | 11.26     | 11.01    |
| 0.66  | 2.3700     | 2.3552    | 10.70     | 10.54    |
| 0.81  | 2.3375     | 2.3242    | 10.35     | 10.22    |
| 0.96  | 2.3094     | 2.2949    | 10.07     | 9.92     |
| 1.11  | 2.2720     | 2.2746    | 9.70      | 9.72     |
| 1.26  | 2.2527     | 2.2533    | 9.51      | 9.52     |
| 1.40  | 2.2345     | 2.2367    | 9.34      | 9.36     |
| 1.55  | 2.2239     | 2.2212    | 9.24      | 9.22     |
| 1.70  | 2.2043     | 2.2044    | 9.06      | 9.06     |
| 1.85  | 2.1885     | 2.1930    | 8.92      | 8.96     |
| 1.99  | 2.1843     | 2.1854    | 8.88      | 8.90     |

## Benchmarks

We benchmarked our model against GPT-2, SmolLM-135M, and Qwen3-0.6B-Base on a question generation task:

| Model | Params | Avg Score | Coherent | Mostly Coherent | Partially Coherent | Incoherent |
|-------|--------|-----------|----------|-----------------|--------------------|------------|
| **StopAskingQuestionsMini** (this) | 656K | 0.4395 | 42 | 60 | 37 | 161 |
| GPT-2 | 117M | 0.3874 | 16 | 50 | 49 | 185 |
| SmolLM2-135M | 135M | 0.5193 | 36 | 98 | 40 | 111 |
| Qwen3-0.6B-Base | 600M | 0.7359 | 165 | 79 | 16 | 40 |

Each model generated two to three hundred continuations of the prefix `Question:`. [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) scored each one using a decimal grading system (0.0 to 1.0).
Our model generated the second highest number of coherent questions with less parameters than most character level RNNs.

## Generations

Prompt: **`Question:`**

Generation1:
```text
what legal reforms faced rafer leadership during ww1?
```

Generation2:
```text
How many emissions should a frather?
```

Generation3:
```text
What do foreigners do?
```

Generation4:
```text
What is the best appropriate way to learn Japanese?
```

Generation5:
```text
How much is the MDU and JavaScript to the new UK?
```

## Use Cases

Unfortunately, there is no practical use case as we stated earlier, but here are some interesting ideas:

1. Test model for pipelines, code, and training
2. Educational research on language models
3. Experimentation on constrained hardware
4. Or, more simply, for fun.

## Limitations

Everything.
But more specifically, 

1. Cannot generate sentences, paragraphs, code, or anything other than questions
2. Cannot reason
3. Short context
4. Incoherent

## Inference

```python
# =============================================================================
#  Inference
# =============================================================================

MODEL_DIR      = "harley-ml/StopAskingQuestionsMini-656k"   # path
TOKENIZER_PATH = "harley-ml/StopAskingQuestionsMini-656k"

# --- Generation settings ---
PROMPT             = "Question:"   # prompt
MAX_NEW_TOKENS     = 96
TEMPERATURE        = 1.0
TOP_P              = 0.95
TOP_K              = 50
REPETITION_PENALTY = 1.1
DO_SAMPLE          = True

# =============================================================================

import torch
from pathlib import Path
from transformers import (
    AutoModelForCausalLM,
    PreTrainedTokenizerFast,
    AddedToken,
)

# ---------------------------------------------------------------------------
# Device
# ---------------------------------------------------------------------------

device = (
    "cuda" if torch.cuda.is_available() else
    "mps"  if torch.backends.mps.is_available() else
    "cpu"
)
print(f"Device : {device}")

# ---------------------------------------------------------------------------
# Tokenizer  (mirrors training setup)
# ---------------------------------------------------------------------------

def load_tokenizer(path: str):
    p = Path(path).resolve()
    if not p.exists():
        raise FileNotFoundError(f"Tokenizer not found: {p}")
    tok = PreTrainedTokenizerFast(tokenizer_file=str(p))
    specials = {}
    if tok.bos_token is None: specials["bos_token"] = AddedToken("<|bos|>", special=True)
    if tok.eos_token is None: specials["eos_token"] = AddedToken("<|eos|>", special=True)
    if tok.unk_token is None: specials["unk_token"] = AddedToken("<|unk|>", special=True)
    if tok.pad_token is None:
        if tok.eos_token is not None:
            tok.pad_token = tok.eos_token
        else:
            specials["pad_token"] = AddedToken("<|pad|>", special=True)
    if specials:
        tok.add_special_tokens(specials)
    tok.padding_side = "left"  # left-pad for batched generation
    return tok

print("Loading tokenizer...")
tokenizer = load_tokenizer(TOKENIZER_PATH)
print(f"  Vocab size : {tokenizer.vocab_size}")
print(f"  BOS        : {tokenizer.bos_token!r}")
print(f"  EOS        : {tokenizer.eos_token!r}")
print(f"  PAD        : {tokenizer.pad_token!r}  (id={tokenizer.pad_token_id})")

# ---------------------------------------------------------------------------
# Model
# ---------------------------------------------------------------------------

print(f"\nLoading model from {MODEL_DIR} ...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_DIR,
    dtype=torch.float16 if device == "cuda" else torch.float32,
    low_cpu_mem_usage=True,
)
model.eval()
model.to(device)

total_params = sum(p.numel() for p in model.parameters())
print(f"  Parameters : {total_params:,}")

# ---------------------------------------------------------------------------
# Generation helper
# ---------------------------------------------------------------------------

def generate(
    prompt: str             = PROMPT,
    max_new_tokens: int     = MAX_NEW_TOKENS,
    temperature: float      = TEMPERATURE,
    top_p: float            = TOP_P,
    top_k: int              = TOP_K,
    repetition_penalty: float = REPETITION_PENALTY,
    do_sample: bool         = DO_SAMPLE,
) -> str:
    
    bos         = tokenizer.bos_token or ""
    full_prompt = bos + prompt

    inputs = tokenizer(
        full_prompt,
        return_tensors="pt",
        add_special_tokens=False,
    ).to(device)
    inputs.pop("token_type_ids", None)  # Qwen3 doesn't use this

    gen_kwargs = dict(
        max_new_tokens     = max_new_tokens,
        do_sample          = do_sample,
        repetition_penalty = repetition_penalty,
        eos_token_id       = tokenizer.eos_token_id,
        pad_token_id       = tokenizer.pad_token_id,
    )
    if do_sample:
        gen_kwargs["temperature"] = temperature
        gen_kwargs["top_p"]       = top_p
        gen_kwargs["top_k"]       = top_k

    with torch.inference_mode():
        output_ids = model.generate(**inputs, **gen_kwargs)

    # Strip the prompt tokens so we only return what was generated
    prompt_len = inputs["input_ids"].shape[-1]
    new_ids    = output_ids[0][prompt_len:]
    return tokenizer.decode(new_ids, skip_special_tokens=True)


# ---------------------------------------------------------------------------
# Run
# ---------------------------------------------------------------------------

if __name__ == "__main__":
    print(f"\nPrompt : {PROMPT!r}")
    print("-" * 60)

    output = generate(PROMPT)

    print("Generated:")
    print(output)
```

## Citation

```bibtex
@misc{stopaskingquestionsmini-656k,
  title     = {StopAskingQuestionsMini-656k: Questions with No Answers},
  author    = {Harley-ml},
  year      = {2026},
  url       = {https://huggingface.co/Harley-ml/StopAskingQuestionsMini-656k}
}
```