README.md

---
license: apache-2.0
tags:
- text-dataset
- pretraining
- llm
- fineweb
- cosmopedia
- math
---

# Base LLM 400M

Self-mined multi-modal pretraining corpus targeting 200B tokens of high-quality text, video, audio, and image data.

## Structure

- `data/shards/shard_{idx:08d}.bin` — raw `uint32` token IDs, ~128MB per shard
- `data/shards/shard_{idx:08d}.meta.json` — 18-field sidecar with quality stats
- `state/` — runtime checkpoints (`.state` extension)

## 18-field sidecar schema

| Field | Type | Description |
|-------|------|-------------|
| `shard_idx` | int | Shard number |
| `filename` | str | Filename |
| `num_tokens` | int | Token count |
| `dtype` | str | `uint32` |
| `size_bytes` | int | File size |
| `created_at` | str | ISO 8601 timestamp |
| `tokens` | int | Token count (duplicate for compatibility) |
| `avg_score` | float | Mean quality score |
| `min_score` | float | Min quality score |
| `max_score` | float | Max quality score |
| `std_score` | float | Std dev of quality scores |
| `n_above_3` | int | Chunks with score > 3.0 |
| `n_above_5` | int | Chunks with score > 5.0 |
| `score_hist` | dict | 21-bin histogram (0.0–10.0 step 0.5) |
| `modality_comp` | dict | Modality composition counts |
| `pillar_comp` | dict | Pillar composition counts |
| `ts` | float | Unix timestamp |
| `modality` | str | Primary modality |

## Loading (Python)

```python
from datasets import load_dataset
ds = load_dataset("morningstarxcdcode/base-llm-400m", split="train", streaming=True)
for example in ds:
    print(example["tokens"])
```