docs/vo_regular_theme_generation.md

# Theme Generation Engines

This note documents the first generation experiments to run after the corpus
audit. There are now two engines sharing the same corpus loader, symbolization,
rhythmic constraints, MIDI/ABC export, and report writer:

- `markov`: a small variable-order Markov model using `vo_regular_bp` regular
  constraints and soft endpoint weights.
- `transformer`: a tiny key-relative decoder-only transformer using the same
  symbolic vocabulary and constrained sampling surface.

The unified runner is:

```bash
python3 scripts/run_theme_generation.py markov
python3 scripts/run_theme_generation.py transformer
```

The old Markov command remains available as a wrapper:

```bash
python3 scripts/run_vo_regular_baseline.py
```

The transformer wrapper is:

```bash
python3 scripts/run_transformer_baseline.py
```

The transformer engine requires PyTorch in the active Python environment. The
local baseline run used:

```bash
python3 -m venv .venv
.venv/bin/pip install torch numpy mido
.venv/bin/python scripts/run_theme_generation.py transformer \
  --samples 6 \
  --length 24 \
  --steps 500 \
  --batch-size 64 \
  --block-size 64 \
  --d-model 96 \
  --nhead 4 \
  --layers 3 \
  --feedforward 192 \
  --temperature 0.95 \
  --top-k 16 \
  --output-dir outputs/transformer_baseline \
  --write-abc
```

## Available Corpus Tables

The generated SQLite database is `audit/themes_audit.sqlite`.

Important tables:

- `themes`: catalog metadata, ABC/MIDI paths, key signature, meter, length, and
  note-level summary features.
- `notes`: one row per MIDI note event, sorted by `start_tick` and
  `note_index`.
- `theme_descriptions` and `theme_text_fts`: local generated search text.
- `endpoint_analysis`: first/last note and first-salient endpoint statistics.
- `key_modulation_analysis`: key-signature coverage, accidental counts, and
  heuristic local-key drift candidates.

## Key Findings Used By The Model

The corpus contains 9,824 parsed themes and 196,679 MIDI note events.

Key metadata:

- Each theme has a single explicit key signature.
- No MIDI file contains multiple key-signature events.
- No ABC file contains inline `[K:...]` key changes.
- Minor mode is not encoded reliably; `abc_key` should be interpreted as a
  key-signature/root feature, not a full tonal label.

Endpoint regularities:

- Last note is the notated tonic/root in only 16.8% of themes.
- Last note is tonic or dominant in 35.3% of themes.
- Last note is in the tonic triad in 54.5% of themes.
- The first-salient-to-last pitch-class relation favors intervals 0, 7, and 5.

Implication: endpoint constraints should be soft priors, not hard cadence rules.

## Initial Symbolization

Start with a deliberately small vocabulary:

```text
symbol = (relative_pitch_class, duration_value)
```

where:

```text
relative_pitch_class = (pitch_class - key_root) mod 12
```

The initial version should keep the original duration labels from `notes`,
possibly filtered to the common values:

- `16th`
- `eighth`
- `quarter`
- `half`
- `dotted eighth`
- `dotted quarter`
- `eighth triplet`
- `16th triplet`

Rare duration labels can either be mapped to the nearest common class or kept
only if they occur above a threshold.

This transposes every theme into a common key-relative space and gives enough
data for variable-order contexts.

The symbolization deliberately removes register. Both the Markov and transformer
engines therefore generate relative pitch classes, not concrete octaves. The
shared output layer realizes generated pitch classes into MIDI pitches for
playback and notation; generated samples currently use a treble-friendly C4-C6
range so ABC, MIDI, MusicXML, and Verovio previews do not drift into unreadable
ledger-line registers. A later model can make register/octave a first-class
prediction target if we want the generators to learn that behavior directly.

## Markov Training Model

Train an order-stack variable-order Markov model over these symbols.

Recommended first settings:

```text
max_order = 4
min_context_count = 2
backoff = longest available context with nonzero feasible future mass
```

The generated sequence length can first be controlled in notes, e.g. 16, 24, or
32 notes. A later version should control length in beats/bars through a duration
tracking acceptor.

## Transformer Training Model

Train a deliberately small decoder-only transformer over the same symbol stream.
The implementation is in `theme_generation/engines/transformer.py` and uses:

```text
token stream = START, symbol_1, symbol_2, ...
symbol = (relative_pitch_class, duration_value)
```

Recommended first settings:

```text
block_size = 64
d_model = 96
nhead = 4
num_layers = 3
steps = 800
top_k = 16
temperature = 1.0
```

This is intentionally small enough to make overfitting visible and iteration
cheap. Once PyTorch is installed, a tiny smoke run is:

```bash
python3 scripts/run_theme_generation.py transformer \
  --samples 2 \
  --length 16 \
  --steps 20 \
  --output-dir outputs/transformer_smoke
```

## Regular / Weighted Constraints

The `vo_regular_bp` acceptor should encode constraints that are finite-state:

1. Allowed symbol vocabulary.
2. Optional maximum repetition of the same relative pitch class.
3. Optional maximum number of chromatic relative pitch classes.
4. Optional accumulated duration / bar length target.
5. Boundary weights for initial and terminal relative degrees.

For the first note-count baseline, the weighted DFA state can be as small as:

```text
state = position
```

and the transition weight can be:

```text
if position == 0:
    return start_degree_weight[relative_pitch_class]
if position == length - 1:
    return terminal_degree_weight[relative_pitch_class]
return 1.0
```

If we want the first-salient-to-last relation, augment the DFA state:

```text
state = (position, first_salient_relative_pc_or_none)
```

and apply the endpoint relation weight on the last transition:

```text
endpoint_weight[first_salient_relative_pc][last_relative_pc]
```

This compiles the non-local relation into regular state memory.

## Soft Priors

Use empirical degree distributions from `endpoint_analysis`, with smoothing.

Start weights:

```text
P(first_salient_degree)
```

End weights:

```text
P(last_degree)
```

Optional relation weights:

```text
P(last_relative_pc | first_salient_relative_pc)
```

These should be temperature-scaled so the Markov model still has room to choose
idiomatic continuations.

## First Experiment

The first runnable Markov experiment:

1. Load `themes` and `notes` from `audit/themes_audit.sqlite`.
2. Build key-relative sequences of `(relative_pitch_class, duration_value)`.
3. Train a max-order Markov count model.
4. Generate samples of fixed note length with endpoint degree soft weights.
5. Convert generated samples to MIDI in a chosen output key.
6. Write a small report with sampled sequences and model diagnostics.

The default generation vocabulary should be rhythmically conservative but not
overly square: allow durations down to a 16th note, require ordinary durations
to fall on a 16th-note grid, and also allow regular triplet durations. This
keeps ordinary short notes, dotted-eighth figures, and eighth-triplets. Triplets
should be constrained to complete beat-aligned groups; isolated or phase-shifted
triplets can make MIDI notation importers quantize surrounding material as
awkward septuplets or nontuplets.
This should be compiled into the finite-state acceptor, using beat phase and
triplet-group state, rather than handled by generate-and-test filtering.

Suggested Markov outputs:

```text
outputs/vo_regular_baseline/
  generated_*.mid
  report.md
```

Suggested transformer outputs:

```text
outputs/transformer_baseline/
  generated_*.mid
  generated_*.abc
  report.md
```

ABC export can still be useful for debugging, but should be opt-in rather than
part of the default generation output.

## Open Questions

- Should length be controlled by note count or by total duration?
- Should rare chromatic degrees be allowed freely, penalized, or modeled as a
  separate chromatic decoration layer?
- Should endpoint priors be global, meter-specific, composer-specific, or
  genre-specific?
- Should local key drift candidates be excluded from training the simplest
  key-relative model?

The simplest useful answer is: start global, note-count based, and
key-relative; then listen.