toolsets/text-transform/doc.md

# text‑transform

Diff computation, regex extraction, Markdown table generation, and text statistics.
Pure‑stdlib toolkit for the text‑munging tasks agents do constantly but currently
need throw‑away scripts for.

---

## When to Use This Toolset

- Comparing two versions of text and getting a unified diff
- Extracting structured data from unstructured text using regex
- Converting tabular data (list of dicts) into Markdown tables for reports
- Computing readability statistics for generated content
- Validating that text transformations produced expected results

---

## Process

### General Text Transformation Workflow

```
Understand the input → Choose the right function → Execute → Validate output
```

1. **Understand** what you're starting with: Is it a single string? Two strings to compare?
   Structured data? Unstructured text with patterns?
2. **Choose** the right function(s) based on the task
3. **Execute** — most functions are one‑shot; combine them for multi‑step transforms
4. **Validate** — use `text_stats` to confirm the output has reasonable characteristics,
   or `text_diff` to verify that a transformation produced expected changes

---

## Function Reference

### `text_diff`

Compute a unified diff between two text blocks.

**When to use:**
- Comparing two versions of a file or output
- Showing exactly what changed after a transformation
- Verifying that an edit only changed what you intended

**Args:**
- `original` (str) — The original text
- `modified` (str) — The modified text
- `context_lines` (int, optional) — Lines of context around changes (default 3)
- `label_a` (str, optional) — Label for original in header
- `label_b` (str, optional) — Label for modified in header

**Returns:** `{diff, added, removed, changed}`

**Workflow tip:** After applying a transformation, use `text_diff(original=before, modified=after)`
to confirm the diff contains only expected changes.

### `regex_extract`

Find all regex matches with positions, capture groups, and named groups.

**When to use:**
- Extracting URLs, emails, dates, or IDs from text
- Parsing structured patterns from semi‑structured output
- Validating that expected patterns exist in generated text

**Args:**
- `text` (str) — The text to search
- `pattern` (str) — Python regex pattern
- `flags` (list, optional) — `["IGNORECASE"]`, `["MULTILINE"]`, `["DOTALL"]`, or combinations
- `max_matches` (int, optional) — Limit matches returned (0 = all)

**Returns:** `{matches: [{index, start, end, text, groups, named_groups?}], count}`

### `markdown_table`

Convert structured data to a formatted Markdown table.

**When to use:**
- Generating report tables from data
- Formatting query results for display
- Creating documentation tables from structured data

**Args:**
- `data` (list) — List of dicts, each dict = one row
- `columns` (list, optional) — Column order (default: keys from first row)
- `align` (str, optional) — `"left"` (default), `"center"`, or `"right"`

**Returns:** `{markdown, rows, columns}`

### `text_stats`

Compute readability and structure statistics for a text block.

**When to use:**
- Checking if generated content is at an appropriate reading level
- Validating that output has expected word/character counts
- Profiling text before processing (e.g., for summarization)

**Args:**
- `text` (str) — The text to analyze

**Returns:** `{chars, words, lines, sentences, paragraphs, avg_word_len, avg_sentence_len, flesch_reading_ease}`

**Flesch Reading Ease scale:**
| Score | Level |
|-------|-------|
| 90–100 | Very easy (5th grade) |
| 60–70 | Plain English (8th–9th grade) |
| 30–50 | College level |
| 0–30 | Very difficult (graduate) |

---

## Common Patterns

### Pattern: Extract and Verify

```
regex_extract → extract all URLs from text
Check count of matches against expectations
If count differs → investigate with text_diff
```

### Pattern: Diff‑Based Validation

```
Save original text
Apply transformation (edit, reformat, translate)
text_diff(original, transformed) → show what changed
Verify only expected changes appear in diff
```

### Pattern: Data → Markdown Report

```
Parse/collect data into list of dicts
markdown_table → formatted table
Embed table in report markdown
text_stats on final report → sanity check
```

### Pattern: Text Quality Check

```
text_stats on generated content
Check flesch_reading_ease — is it appropriate for the audience?
Check avg_sentence_len — too long? (>25 words = complex)
Adjust content and re‑check
```

---

## Guidelines

### Do
- Use `text_diff` after any transformation to verify changes are expected
- Anchor regex patterns with `^` and `$` when matching complete lines
- Use `max_matches` for large texts to avoid overwhelming output
- Check `text_stats` on output to catch structural issues (e.g., all text collapsed to one line)
- Use `regex_extract` with `flags=["IGNORECASE"]` when case doesn't matter

### Don't
- Don't use regex to parse nested structures (HTML, JSON, XML) — use proper parsers
- Don't trust regex patterns without testing on edge cases first
- Don't generate markdown tables with inconsistent column counts across rows
- Don't treat Flesch score as absolute — it's a rough heuristic, not a quality guarantee
- Don't `text_diff` on very large files (10k+ lines) — use `context_lines=0` or compare smaller chunks

### Regex Pattern Pitfalls
- **Greedy quantifiers**: `.*` matches too much — use `.*?` for non‑greedy
- **Dot matches newline**: Use `flags=["DOTALL"]` if you want `.` to match `\n`
- **Unescaped special chars**: Escape `.`, `*`, `+`, `?`, `[`, `]`, `(`, `)`, `{`, `}`, `|`, `^`, `$`
- **Character classes**: `\d` matches Unicode digits too — use `[0-9]` for ASCII‑only