README.md

---
language:
- en
license: apache-2.0
pipeline_tag: text-generation
library_name: llama-cpp-python
datasets:
- >-
  shreyan35/Project-Axiom-1.0-Opus4.7-Kimi2.6-GLM5.2-Deepseek4-Mythos5-Fable5-Qwen3.7
tags:
- gguf
- qwen3
- causal-lm
- small-language-model
- text-generation
---

# Flare-50M

**Flare-50M** is a compact 49M-parameter causal language model distributed in
GGUF format. It is intended for research, experimentation, local inference, and
studying the capabilities and limitations of very small language models.

The included checkpoint uses Q8_0 quantization and is approximately 53 MB.

## Model details

| Property | Value |
|---|---:|
| Architecture | Qwen3-style causal LM |
| Parameters | 49M (approximately 50M) |
| Layers | 12 |
| Hidden size | 512 |
| Attention heads | 8 |
| Key/value heads | 2 |
| Intermediate size | 1,344 |
| Vocabulary size | 32,000 |
| Maximum context | 1,024 tokens |
| Quantization | GGUF Q8_0 |
| Checkpoint | `50m-q8_0.gguf` |

## Intended use

Flare-50M may be useful for:

- experiments with highly constrained language models;
- local CPU inference;
- language-model training and evaluation research;
- testing compression, quantization, and fine-tuning pipelines;
- basic English completion and syntax experiments.

The model is not intended to be a general-purpose assistant or a reliable
source of factual information.

## Training data

Flare-50M was trained using data from:

- [Project Axiom 1.0 — Opus 4.7, Kimi 2.6, GLM 5.2, DeepSeek 4, Mythos 5,
  Fable 5, and Qwen 3.7](https://huggingface.co/datasets/shreyan35/Project-Axiom-1.0-Opus4.7-Kimi2.6-GLM5.2-Deepseek4-Mythos5-Fable5-Qwen3.7)

Project Axiom is an English synthetic reasoning and instruction corpus containing
chain-of-thought traces, multi-turn conversations, mathematical reasoning, code,
tool trajectories, and other structured problem-solving examples. The dataset
repository is published under the MIT license.

The exact subset, number of training tokens, preprocessing procedure, and number
of epochs used for Flare-50M have not been documented in this model card. The
dataset contains synthetic model-generated material, so its outputs may include
incorrect reasoning, factual errors, stylistic artifacts, and biases inherited
from the teacher models.

## Usage

Install a recent version of `llama-cpp-python`. Older releases may not recognize
the Qwen3 GGUF architecture.

```bash
pip install "llama-cpp-python>=0.3.34"
```

```python
from llama_cpp import Llama

model = Llama(
    model_path="50m-q8_0.gguf",
    n_ctx=1024,
    verbose=False,
)

result = model(
    "The future of small language models is",
    max_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repeat_penalty=1.1,
)

print(result["choices"][0]["text"])
```

Do not set `n_ctx` above 1,024 unless you are intentionally testing unsupported
context extrapolation.

## Evaluation

The following results were measured on a fixed random sample of 1,000 examples
per task. WikiText-2 used approximately 100,000 tokens. Candidate-based tasks
were evaluated using conditional continuation log-likelihood instead of asking
the model to generate an answer letter.

| Benchmark | Metric | Flare-50M | Random baseline |
|---|---|---:|---:|
| WikiText-2 | Perplexity ↓ | 340.38 | — |
| LAMBADA OpenAI | Last-word accuracy ↑ | 7.5% | — |
| BLiMP | Accuracy ↑ | 61.8% | 50% |
| HellaSwag | Accuracy ↑ | 25.8% | 25% |
| PIQA | Accuracy ↑ | 53.3% | 50% |
| ARC-Easy | Accuracy ↑ | 29.9% | approximately 25% |

Additional length-normalized scores:

| Benchmark | Normalized accuracy |
|---|---:|
| BLiMP | 59.0% |
| HellaSwag | 25.8% |
| PIQA | 52.8% |
| ARC-Easy | 30.5% |

### Evaluation notes

- These results were produced with a custom `llama-cpp-python` evaluation
  script and may not be directly comparable to results from the EleutherAI
  LM Evaluation Harness or public leaderboards.
- Perplexity is tokenizer-dependent and should only be compared under the same
  tokenization and evaluation procedure.
- BLiMP is the clearest positive result and suggests that the model learned some
  English syntactic structure.
- HellaSwag is at approximately random performance. PIQA and ARC-Easy show only
  small improvements over their respective chance levels.
- A direct zero-shot MMLU answer-letter test was not considered valid: the model
  selected `A` for every item, demonstrating strong answer-token bias rather
  than measurable multiple-choice knowledge.

## Limitations

Flare-50M is an extremely small model by modern standards. Users should expect:

- limited factual knowledge;
- weak multi-step reasoning and instruction following;
- poor performance on long or complex prompts;
- repetition, contradictions, and incoherent generations;
- a hard context limit of 1,024 tokens;
- biases and inaccuracies inherited from its training data.

Model outputs should not be used for medical, legal, financial, safety-critical,
or other high-stakes decisions.

## Reproducibility

Evaluation configuration:

```text
seed: 1234
quantization: Q8_0
context length: 1024
examples per task: 1000
WikiText-2 evaluated tokens: 99903
scoring: conditional continuation log-likelihood
```

Dataset splits:

- WikiText-2 Raw: `test`
- LAMBADA OpenAI: `test`
- BLiMP: benchmark `train` splits (the dataset's evaluation examples)
- HellaSwag: `validation`
- PIQA: `validation`
- ARC-Easy: `test`