README.md

---
license: mit
datasets:
- roneneldan/TinyStories
language:
- en
pipeline_tag: text-generation
---
# Test-1-4000 — A 190M Parameter Narrative Engine

---

## Overview

Test-1-4000 is the final training checkpoint of a compact decoder-only Transformer model built on the Llama architecture and trained on the TinyStories dataset.

The project focuses on studying how narrative coherence, logical consistency, and language fluency emerge inside small-scale language models through structured training.

By Step 4000, the model reaches a significantly higher level of generative stability and narrative fluency compared to earlier checkpoints, achieving a final training loss of **0.573** after nearly two full epochs of training.

---

## Model Highlights

| Feature | Specification |
|---|---|
| Architecture | Llama-based Decoder-only Transformer |
| Parameters | 190.55 Million |
| Context Window | 2048 Tokens |
| Final Training Step | 4000 |
| Final Training Loss | 0.573 |
| Precision | bfloat16 |
| Attention Backend | Flash Attention 2 |
| Compilation | torch.compile |
| Tokenizer | GPT-2 Tokenizer |

---

## Architecture

| Component | Value |
|---|---|
| Hidden Dimension | 768 |
| Layers | 12 |
| Attention Heads | 12 |
| Intermediate Size | 3072 |
| Activation Function | SwiGLU |
| Normalization | RMSNorm |
| Vocabulary Size | 50,257 |

The model uses Rotary Positional Embeddings (RoPE) for stable long-range token relationships across the 2048-token context window.

---

## Training Progression

### Phase 1 — Lexical Learning (0 → 250)

The model learned grammar, sentence formation, and common linguistic patterns.

### Phase 2 — Relational Understanding (250 → 1000)

The model began associating entities, actions, and environments into logically connected sequences.

### Phase 3 — Narrative Coherence (1000 → 2000)

Narrative continuity emerged. Stories developed stable structure, conflict resolution, and reduced contradiction.

### Phase 4 — Emergent Narrative Intelligence (2000 → 3000)

The model improved in emotional consistency, long-range memory, and thematic continuity across generations.

### Phase 5 — Fluent Generative Stability (3000 → 4000)

This final phase marked a transition from structured storytelling into fluent narrative generation.

The model became substantially better at:

- maintaining tone,
- producing natural sentence flow,
- avoiding repetitive degeneration,
- preserving character consistency,
- and generating smoother transitions between events.

By this stage, generations began feeling less mechanically predicted and more organically written. Dialogue improved noticeably, pacing became more natural, and narrative structure stabilized across longer outputs.

The reduction in loss to **0.573** indicates a major improvement in predictive confidence and language fluency.

---

## Training Configuration

| Parameter | Value |
|---|---|
| Optimizer | AdamW |
| Learning Rate | 5e-4 |
| Scheduler | OneCycleLR |
| Weight Decay | 0.01 |
| Precision | bfloat16 |
| Effective Batch Size | ~262K tokens/step |

---

## Dataset

The model was trained on TinyStories, a synthetic storytelling dataset designed to teach language models reasoning and narrative structure using simplified vocabulary and clean writing patterns.

This allows the model to focus on:

- causal reasoning,
- narrative flow,
- emotional continuity,
- and long-range coherence.

---

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "GODELEV/Test-1-4000"

tokenizer = AutoTokenizer.from_pretrained(model_path)

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "Once upon a time, a boy found a silver key."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id
)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

---

## Final Notes

Test-1-4000 demonstrates that coherent and fluent narrative behavior can emerge in compact Transformer models when training is focused on clean, structured data and long-form consistency.

Despite its relatively small size, the model exhibits:

- strong narrative fluency,
- stable story progression,
- coherent emotional structure,
- and reliable long-context generation.

The project serves as an exploration into how efficient language models can develop increasingly sophisticated generative behavior through progressive training refinement.

---

## Citation

```bibtex
@misc{test14000,
  title={Test-1-4000: A 190M Parameter Narrative Engine},
  author={GODELEV},
  year={2026}
}
```