skills/codex_extractor/SKILL.md

# TOTEM Studio Skill — Codex Extractor

## Purpose
Use this skill whenever editing, reviewing, testing, or prompting work on `src/codex_extractor.py`.

The Codex Extractor is a developer-only tool. It extracts raw numerical and categorical author-style fingerprint data from legally accessed PDF/text samples. It does not create final Codex fingerprints by itself.

## Core Terminology
- **TOTEM** = manuscript auditing layer.
- **Codex** = author-style comparison and co-creation layer.
- **Codex Extractor** = developer-only extraction tool that produces raw VM metric data.
- **Final Codex Fingerprint** = validated/aggregated author model stored in Codex after QA.

## Current Governance State
- **VM-001 to VM-005 are frozen under Spec v1 calibrated.**
- **Spec v1 controlled validation is closed and accepted.**
- Further VM-001 to VM-005 changes require a formal approved **Spec v1.1 issue**.
- Do not tune VM-004 or any VM-001 to VM-005 rule during normal patching, validation, or sample preparation.
- Validation failures must be classified first as one of:
  - data/sample issue
  - traceability issue
  - runtime issue
  - potential Spec v1.1 candidate
- Only a confirmed, approved Spec v1.1 candidate may lead to VM-001 to VM-005 logic changes.

## Non-Negotiable Rules
1. Never store copyrighted source text in final Codex fingerprints.
2. Store only derived values: numbers, labels, confidence scores, QA flags, provenance metadata, and extractor version.
3. Raw extractor output is not automatically a final fingerprint.
4. QA/debug artefacts may contain source fragments, but they are developer-only and must not enter final fingerprint storage.
5. Preserve existing public function names unless explicitly instructed otherwise:
   - `process_upload()`
   - `extract_fingerprint()`
   - `format_fingerprint_report()`
   - `extract_text_from_file()`
6. Do not redesign the extractor when asked for a patch.
7. Patch only the requested subsystem.

## Patch Discipline
When modifying `codex_extractor.py`:
- Make the smallest possible change.
- Do not alter unrelated VM metrics.
- Do not touch OCR, page filtering, metadata provenance, dialogue, repetition, or workbook output unless the user specifically requests it.
- Do not change VM-006 to VM-013 or VM-024 to VM-028 unless explicitly instructed.
- Do not change Tier 2 handling.
- Do not infer bibliography or works sampled. `Works_Sampled` must reflect user input or the actual uploaded filename only.
- Do not change VM-001 to VM-005 unless the user explicitly approves a formal Spec v1.1 issue.

## VM-001 to VM-005 Standard
VM-001 to VM-005 must follow **TOTEM Codex Rhyme & Prosody Algorithm Specification v1**.

This includes:
- syllable counting with CMUdict first, fallback heuristic second
- token-level syllable confidence
- stress pattern extraction from ARPAbet where available
- dominant metre/foot support metrics
- phonetic-first rhyme detection
- rhyme-tail comparison from the last stressed vowel onward
- fallback handling for unknown, invented, proper-noun, and OCR-damaged words
- weighted rhyme density
- guarded rhyme scheme classification
- `FREE` must not be returned when meaningful rhyme evidence exists

## Spec v1 Controlled Validation Status
Spec v1 controlled validation has been accepted and closed.

Closure facts:
- controlled sample pack validation completed
- final targeted rerun passed
- no runtime issues remained
- no traceability gaps remained
- no extractor logic changes were needed during closure
- no VM-001 to VM-005 tuning was performed
- no Spec v1.1 issue was required

Governance consequence:
- Treat VM-001 to VM-005 as stable under Spec v1.
- Do not reopen VM-001 to VM-005 logic because of a single surprising sample.
- First review sample quality, expected labels, rights/provenance metadata, and debug artefacts.
- Only escalate to Spec v1.1 if the failure is reproducible, traceable, not caused by data/sample construction, and approved by the validation lead.

## Manual Page Override Rule
Manual story-page range must beat automatic filtering.

If `start_page` or `end_page` is provided:
- process only pages inside that inclusive range
- mark pages outside the range as `out_of_range`
- do not auto-skip the first page if it is inside the manual range
- do not apply front-matter exclusion to pages inside the manual range
- record manual override metadata in `page_trace.csv`, `metric_trace.json`, and the output report

If no manual range is provided:
- use automatic page filtering
- raise a QA warning if OCR is used without a manual story-page range

## Required Debug Artefacts
Every extraction run should be able to produce:
- `cleaned_text.txt`
- `page_trace.csv`
- `line_endings.csv`
- `rhyme_pairs_debug.csv`
- `syllable_debug.csv`
- `stress_pattern_debug.csv`
- `unknown_tokens.csv`
- `repetition_matches.csv`
- `prosody_summary.json` where implemented

`prosody_summary.json` must not contain source text.

## QA Expectations
Always test in this order:
1. clean text sample
2. controlled Gruffalo-style rhymed sample
3. scanned/image-only PDF with manual page range
4. scanned/image-only PDF with automatic filtering

Expected behaviour for clear rhymed picture-book verse:
- VM-004 must not return `FREE`
- VM-003 must show meaningful rhyme evidence
- `rhyme_pairs_debug.csv` must prove the classification
- OCR uncertainty must reduce confidence, not hide uncertainty

## Controlled Validation Discipline
For controlled validation or reruns:
- Do not alter sample text during a run.
- Do not alter expected labels during a run.
- Do not tune extractor logic during validation.
- Preserve per-sample artefacts.
- Preserve run-level metadata, reviewer notes, pass/fail ledger, and checklist.
- If a sample fails, classify the failure before proposing any action.
- Prefer data correction, label correction, retirement, or replacement before any Spec v1.1 escalation.

## Codex Prompting Rule
When using ChatGPT Codex or another coding agent, tell it:

> Before editing, read `skills/codex_extractor/SKILL.md`. Obey it. Patch only the requested subsystem. Do not redesign unrelated code. VM-001 to VM-005 are frozen under Spec v1 unless I approve a formal Spec v1.1 issue.

## Escalation Rule
If a requested change conflicts with this skill:
- stop
- explain the conflict
- ask for explicit override before proceeding