---
title: Pre-Punctuation Processor
emoji: 📜
colorFrom: yellow
colorTo: gray
sdk: gradio
app_file: app.py
pinned: false
license: mit
tags:
  - philosophy
  - nlp
  - training-data
  - classical-texts
  - character-level
---

# Pre-Punctuation Processor

A text processing pipeline that prepares ancient philosophical texts as training data for character-level language models, stripping them back to a pre-punctuation form faithful to how they were originally composed and spoken.

## Why Pre-Punctuation?

The philosophical texts in this corpus — Aristotle, Plato, Euclid, Seneca, Epictetus, Marcus Aurelius — were composed in an era before modern punctuation existed. Ancient Greek was written in *scriptio continua*: an unbroken stream of uppercase letters with no spaces, no commas, no quotation marks, no paragraph breaks.

The first systematic punctuation was invented by **Aristophanes of Byzantium** (c. 257–185 BC), head librarian of the Library of Alexandria. He devised a system of single dots (*théseis*) placed at different heights to mark breathing pauses for readers:

- **stigmḕ mésē** (·) mid-level dot — a short pause (*komma*)
- **hypostigmḗ** (.) low dot — a medium pause (*kolon*)
- **stigmḕ teleía** (˙) high dot — a full stop (*periodos*)

This system was a reading aid, not part of the texts themselves. The words of the philosophers predated any notation for pauses or structure.

## The Period as Pause Marker

This pipeline reduces all punctuation to a single mark: the **period** — a direct descendant of Aristophanes' dot system. In our output, the period functions not as a grammatical construct but as what it originally was: a marker for a pause in speech.

The resulting vocabulary is exactly **28 characters**: the 26 lowercase Latin letters, a space, and a period.

## What This Tool Does

1. **Strips all non-body content** — Prefaces, editor's notes, appendixes, transcriber corrections, publisher info, and source boilerplate (Gutenberg, MIT Classics, Internet Archive) are aggressively removed. Only the philosopher's own words remain.
2. **Converts numerals to words** — Both Arabic (600 → "six hundred") and Roman (XIV → "fourteen") numerals become English words.
3. **Normalizes to 28-char vocabulary** — Unicode normalized to ASCII, lowercased, all punctuation except period removed.
4. **Chunks for training** — Text split into 40–256 character chunks at sentence boundaries.
5. **Publishes to HuggingFace** — Train/validation splits pushed as a dataset for direct loading in notebooks.

## Usage

**Drag and drop** a .txt, .epub, or .zip file, or paste a URL from Project Gutenberg, MIT Internet Classics, or the Internet Archive. The pipeline processes it and adds it to the corpus.

**Search the Internet Archive** to browse and add classical texts directly.

**Push to HuggingFace** to make the dataset available anywhere:

```python
from datasets import load_dataset
ds = load_dataset("LisaMegaWatts/philosophy-corpus")
```

## Built for JuliaGPT

The output is designed for training a character-level GPT implemented in Julia, with a target vocabulary of 29 tokens (28 characters + BOS).